Nov 04 00:56:19.736590 ip-10-0-136-94 systemd[1]: Starting Kubernetes Kubelet... Nov 04 00:56:20.152628 ip-10-0-136-94 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:56:20.152628 ip-10-0-136-94 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 00:56:20.152628 ip-10-0-136-94 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:56:20.152628 ip-10-0-136-94 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:56:20.156662 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156580 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:20.156662 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156661 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:20.156662 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156665 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156668 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156673 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156676 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156678 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156681 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156686 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156688 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156691 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156694 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156697 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156705 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156708 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156710 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156714 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156716 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156719 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156721 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156724 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:20.156757 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156726 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156729 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156731 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156734 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156737 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156739 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156742 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156744 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156747 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156750 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156752 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156754 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156757 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156760 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156763 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156766 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156769 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156772 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156775 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156777 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:20.157190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156780 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156782 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156785 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156787 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156791 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156793 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156796 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156798 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156801 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156803 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156806 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156808 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156811 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156814 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156817 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156820 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156823 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156825 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156828 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156831 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:20.157679 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156833 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156836 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156838 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156841 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156843 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156847 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156852 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156855 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156858 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156861 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156864 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156867 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156870 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156872 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156875 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156877 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156880 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156883 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156886 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156888 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:20.158141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156891 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156894 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156896 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156899 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156901 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156904 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156909 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156913 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156917 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156921 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156924 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156926 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156929 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156932 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156934 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156937 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156939 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156942 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156944 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:20.158665 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156947 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156950 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156953 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156955 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156958 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.156960 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157745 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157752 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157755 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157758 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157760 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157764 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157767 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157770 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157772 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157775 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157778 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157781 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157783 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157787 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:20.159190 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157790 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157792 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157795 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157798 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157800 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157803 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157805 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157808 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157811 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157814 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157816 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157819 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157821 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157824 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157827 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157830 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157832 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157835 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157838 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157840 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:20.159676 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157843 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157846 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157848 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157851 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157854 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157857 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157859 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157861 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157864 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157867 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157869 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157872 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157875 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157877 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157880 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157883 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157885 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157888 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157890 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157893 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:20.160143 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157896 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157898 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157901 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157904 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157906 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157908 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157912 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157916 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157919 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157922 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157925 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157928 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157930 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157933 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157936 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157938 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157941 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157944 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:20.160647 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157949 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157954 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157958 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157961 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157964 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157966 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157970 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157973 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157976 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157979 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157982 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157984 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157987 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157989 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157992 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157995 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.157997 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158000 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158003 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158006 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:20.161098 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158008 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158011 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158013 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158016 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158018 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158021 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158023 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158026 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158029 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158032 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158034 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158037 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158040 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158042 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158548 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158555 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158559 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158562 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158565 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:20.161575 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158567 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158570 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158573 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158576 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158578 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158581 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158584 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158587 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158589 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158592 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158595 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158597 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158600 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158603 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158605 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158609 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158612 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158616 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158619 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:20.162018 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158622 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158624 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158627 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158630 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158632 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158635 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158638 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158641 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158643 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158646 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158651 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158654 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158656 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158659 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158662 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158665 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158667 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158670 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158672 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158675 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:20.162467 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158678 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158680 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158683 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158686 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158688 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158691 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158693 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158696 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158698 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158701 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158703 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158706 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158708 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158711 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158713 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158716 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158718 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158721 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158724 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158727 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:20.162934 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158729 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158732 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158736 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158739 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158741 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158745 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158748 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158751 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158753 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158756 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158758 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158761 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158763 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158766 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158770 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158774 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158777 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158780 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158783 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:20.163426 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158785 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158788 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158790 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158793 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158795 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158797 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158800 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158802 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158805 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158807 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158810 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158813 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158816 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158819 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158821 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158825 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158828 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158830 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158834 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158836 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:20.163871 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158839 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158841 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.158844 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159828 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159840 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159846 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159851 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159856 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159860 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159864 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159868 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159872 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159875 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159878 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159882 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159885 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159888 2577 flags.go:64] FLAG: --cgroup-root="" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159891 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159894 2577 flags.go:64] FLAG: --client-ca-file="" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159896 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159899 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159904 2577 flags.go:64] FLAG: --cluster-domain="" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159906 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 00:56:20.164356 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159910 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159913 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159917 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159921 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159927 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159930 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159934 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159938 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159941 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159944 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159948 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159951 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159955 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159958 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159961 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159964 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159967 2577 flags.go:64] FLAG: --enable-server="true" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159970 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159974 2577 flags.go:64] FLAG: --event-burst="100" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159977 2577 flags.go:64] FLAG: --event-qps="50" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159981 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159984 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159987 2577 flags.go:64] FLAG: --eviction-hard="" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159991 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 00:56:20.164906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159995 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.159998 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160001 2577 flags.go:64] FLAG: --eviction-soft="" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160005 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160008 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160011 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160014 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160017 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160020 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160023 2577 flags.go:64] FLAG: --feature-gates="" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160027 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160030 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160035 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160039 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160042 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160045 2577 flags.go:64] FLAG: --help="false" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160048 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160051 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160055 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160058 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160061 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160064 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160068 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 00:56:20.165520 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160071 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160074 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160076 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160079 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160083 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160086 2577 flags.go:64] FLAG: --kube-reserved="" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160089 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160092 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160095 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160098 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160101 2577 flags.go:64] FLAG: --lock-file="" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160104 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160107 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160110 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160115 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160118 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160121 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160124 2577 flags.go:64] FLAG: --logging-format="text" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160127 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160130 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160133 2577 flags.go:64] FLAG: --manifest-url="" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160138 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160143 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160146 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160150 2577 flags.go:64] FLAG: --max-pods="110" Nov 04 00:56:20.166059 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160153 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160156 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160159 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160162 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160165 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160168 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160171 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160178 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160182 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160185 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160188 2577 flags.go:64] FLAG: --pod-cidr="" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160191 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160194 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160197 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160200 2577 flags.go:64] FLAG: --port="10250" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160203 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160207 2577 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-07eb3e70498f5ac31" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160210 2577 flags.go:64] FLAG: --qos-reserved="" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160213 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160216 2577 flags.go:64] FLAG: --register-node="true" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160219 2577 flags.go:64] FLAG: --register-with-taints="" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160223 2577 flags.go:64] FLAG: --registry-burst="10" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160226 2577 flags.go:64] FLAG: --registry-qps="5" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160229 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160232 2577 flags.go:64] FLAG: --reserved-memory="" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160236 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 00:56:20.166696 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160239 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160242 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160246 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160249 2577 flags.go:64] FLAG: --runonce="false" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160253 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160256 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160259 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160262 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160265 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160268 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160271 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160274 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160277 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160280 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160282 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160285 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160288 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160292 2577 flags.go:64] FLAG: --system-cgroups="" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160294 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160300 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160306 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160309 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160324 2577 flags.go:64] FLAG: --tls-min-version="" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160327 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160330 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 00:56:20.167303 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160333 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160335 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160338 2577 flags.go:64] FLAG: --v="2" Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160343 2577 flags.go:64] FLAG: --version="false" Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160347 2577 flags.go:64] FLAG: --vmodule="" Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160351 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160354 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160487 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160491 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160494 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160497 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160499 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160503 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160507 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160511 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160513 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160516 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160519 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160536 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160540 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:20.167892 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160543 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160546 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160548 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160553 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160556 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160559 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160561 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160565 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160574 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160577 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160580 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160583 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160586 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160589 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160591 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160594 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160597 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160599 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160602 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:20.168430 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160604 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160607 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160610 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160613 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160616 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160618 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160622 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160625 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160627 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160630 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160633 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160635 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160638 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160640 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160643 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160645 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160648 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160651 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160653 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160656 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:20.169213 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160660 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160663 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160665 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160668 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160670 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160673 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160676 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160678 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160681 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160683 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160685 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160688 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160691 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160693 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160696 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160699 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160702 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160704 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160707 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:20.169887 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160710 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160713 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160716 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160718 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160721 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160723 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160725 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160728 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160731 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160733 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160736 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160738 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160741 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160745 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160747 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160750 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160752 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160755 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160757 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160760 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:20.170498 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160762 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160765 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160767 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160770 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160772 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160775 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160777 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160781 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160783 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160786 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160789 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160791 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160794 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160797 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160800 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.160826 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:56:20.170975 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160861 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160865 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160868 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160871 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160874 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160877 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160880 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160882 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160885 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160889 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160891 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160894 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160896 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160899 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160902 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160904 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160907 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160909 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160912 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160914 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:20.171373 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160917 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160919 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160922 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160924 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160928 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160931 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160933 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160936 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160938 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160941 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160944 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160947 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160950 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160952 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160955 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160958 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160960 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160963 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160965 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160967 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:20.171838 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160971 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160975 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160977 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160980 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160982 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160985 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160987 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160990 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160993 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160995 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.160998 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161002 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161005 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161009 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161011 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161014 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161018 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161021 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161023 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:20.172311 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161026 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161028 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161033 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161036 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161040 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161043 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161046 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161048 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161051 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161053 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161056 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161058 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161061 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161064 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161068 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161071 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161073 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161076 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161078 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:20.172764 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161081 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161083 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161086 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161088 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161091 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161093 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161096 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161098 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161101 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161104 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161107 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161110 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161113 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161116 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161119 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161122 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161124 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161126 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161129 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161133 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:20.173210 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161135 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161138 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161140 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161143 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161145 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161148 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161150 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.161154 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.170014 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.170033 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.170697 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.170704 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170766 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170771 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170775 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170778 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170781 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170784 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170787 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:20.173692 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170790 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170793 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170796 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170799 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170802 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170805 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170808 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170810 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170813 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170816 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170818 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170821 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170824 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170827 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170829 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170832 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170835 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170838 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170841 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170843 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:20.174141 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170846 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170849 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170852 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170855 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170858 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170861 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170864 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170866 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170869 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170872 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170874 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170879 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170882 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170885 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170888 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170890 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170893 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170896 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170898 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:20.174628 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170902 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170905 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170908 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170910 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170913 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170916 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170919 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170921 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170924 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170926 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170929 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170931 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170934 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170937 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170940 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170943 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170946 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170948 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170951 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170954 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:20.175062 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170957 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170959 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170962 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170965 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170967 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170970 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170972 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170975 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170978 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170981 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170983 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170986 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170989 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170993 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170996 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.170998 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171001 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171004 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171006 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171009 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:20.175552 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171012 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171014 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171017 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171019 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171022 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171025 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171027 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171030 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171033 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171036 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171038 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171041 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171044 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171047 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171049 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171053 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171057 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171059 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171062 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:20.176043 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171065 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.171095 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171255 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171261 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171264 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171268 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171271 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171274 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171276 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171279 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171282 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171284 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171287 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171290 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171292 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171295 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:20.176493 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171297 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171300 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171302 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171305 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171308 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171324 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171329 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171332 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171335 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171338 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171341 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171343 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171346 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171349 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171351 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171354 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171357 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171360 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171362 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171365 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:20.176900 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171368 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171370 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171373 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171377 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171381 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171386 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171389 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171391 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171394 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171396 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171399 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171402 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171404 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171407 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171409 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171412 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171415 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171417 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171420 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:20.177383 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171422 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171425 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171427 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171430 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171432 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171435 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171437 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171440 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171442 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171445 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171447 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171450 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171453 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171455 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171458 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171460 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171463 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171465 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171469 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:20.177811 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171472 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171475 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171477 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171480 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171482 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171485 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171487 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171489 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171493 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171497 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171500 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171503 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171505 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171508 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171511 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171513 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171515 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171518 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171521 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:20.178256 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171523 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171526 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171528 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171530 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171533 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171535 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171538 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171540 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171543 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171545 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171548 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171551 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171553 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171557 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171559 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.171582 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:56:20.178738 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171614 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171618 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171621 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171624 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171627 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171630 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171633 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171636 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171638 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171641 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171643 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171646 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171648 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171651 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171654 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171656 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171659 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171661 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171664 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:20.179112 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171666 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171669 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171671 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171675 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171678 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171681 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171684 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171687 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171690 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171692 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171694 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171698 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171700 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171703 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171705 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171708 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171710 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171713 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171715 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:20.179570 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171718 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171720 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171723 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171725 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171728 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171730 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171733 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171735 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171738 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171740 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171743 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171746 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171748 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171751 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171753 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171756 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171758 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171760 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171763 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:20.180006 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171768 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171771 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171774 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171776 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171779 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171781 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171786 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171788 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171792 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171794 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171797 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171799 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171802 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171805 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171807 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171809 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171812 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171815 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171817 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171820 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:20.180454 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171822 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171825 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171828 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171831 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171833 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171836 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171839 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171841 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171845 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171849 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171853 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171855 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171860 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171862 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171865 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171867 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171871 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171873 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171876 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171879 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:20.180953 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171882 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171884 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171887 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171889 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171892 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171894 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171897 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171900 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:20.171903 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.171961 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.174147 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.174971 2577 server.go:1014] "Starting client certificate rotation" Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.175061 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.175780 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.176842 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:56:20.181428 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.177946 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:56:20.188948 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.188932 2577 log.go:25] "Validated CRI v1 runtime API" Nov 04 00:56:20.192262 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.192249 2577 log.go:25] "Validated CRI v1 image API" Nov 04 00:56:20.193046 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.193026 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 00:56:20.197640 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.197621 2577 fs.go:135] Filesystem UUIDs: map[11e46120-6bcf-4b2d-b33b-787e37952bef:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 97c4d611-aa15-459d-9dfa-f8aca5eab90f:/dev/nvme0n1p3] Nov 04 00:56:20.197686 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.197641 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 00:56:20.204556 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.204437 2577 manager.go:217] Machine: {Timestamp:2026-11-04 00:56:20.201679032 +0000 UTC m=+0.345730765 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3065179 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23bb6f8d874a9138b8d9423342e218 SystemUUID:ec23bb6f-8d87-4a91-38b8-d9423342e218 BootID:d3c05ee4-d1de-42e4-8666-820d5fc407cc Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d3:84:3f:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d3:84:3f:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:f6:c8:11:5e:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 00:56:20.204556 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.204555 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 00:56:20.204660 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.204645 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 00:56:20.207571 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.207551 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 00:56:20.207709 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.207572 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-94.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 00:56:20.208550 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.208539 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 00:56:20.208590 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.208553 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 00:56:20.208590 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.208585 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:56:20.208640 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.208598 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:56:20.209385 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.209374 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 00:56:20.212183 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.212172 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 00:56:20.212340 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.212330 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 00:56:20.212918 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.212909 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 04 00:56:20.212953 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.212925 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 00:56:20.212981 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.212957 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 00:56:20.212981 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.212967 2577 kubelet.go:405] "Adding apiserver pod source" Nov 04 00:56:20.212981 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.212975 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 00:56:20.213901 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.213876 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:56:20.219135 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.219116 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 00:56:20.224801 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.224785 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:56:20.225604 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.225590 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 00:56:20.225667 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.225630 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 00:56:20.225832 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.225817 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 00:56:20.225898 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.225850 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 00:56:20.225898 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.225863 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 00:56:20.225898 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.225878 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 00:56:20.225898 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.225888 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 00:56:20.225898 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.225898 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 00:56:20.226402 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.226392 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 00:56:20.226464 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.226408 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 00:56:20.226464 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.226420 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 00:56:20.226464 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.226431 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 00:56:20.226464 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.226444 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 00:56:20.226464 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.226458 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 00:56:20.227102 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.227091 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 00:56:20.227149 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.227106 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 00:56:20.230766 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.230751 2577 server.go:1263] "Started kubelet" Nov 04 00:56:20.230846 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.230817 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 00:56:20.230950 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.230908 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 00:56:20.230990 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.230969 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 00:56:20.231921 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.231899 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 00:56:20.232356 ip-10-0-136-94 systemd[1]: Started Kubernetes Kubelet. Nov 04 00:56:20.234528 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.234511 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 04 00:56:20.238783 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:20.238120 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 00:56:20.238936 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.238863 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 00:56:20.238936 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.238886 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 00:56:20.239053 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:20.239007 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-94.us-west-2.compute.internal\" not found" Nov 04 00:56:20.239673 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.239592 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 00:56:20.239673 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.239598 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 00:56:20.239673 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.239632 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 00:56:20.239858 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.239688 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 04 00:56:20.239858 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.239695 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 04 00:56:20.240307 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.240293 2577 factory.go:153] Registering CRI-O factory Nov 04 00:56:20.240393 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.240333 2577 factory.go:223] Registration of the crio container factory successfully Nov 04 00:56:20.240393 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.240389 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 00:56:20.240497 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.240399 2577 factory.go:55] Registering systemd factory Nov 04 00:56:20.240497 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.240406 2577 factory.go:223] Registration of the systemd container factory successfully Nov 04 00:56:20.240497 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.240432 2577 factory.go:103] Registering Raw factory Nov 04 00:56:20.240497 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.240445 2577 manager.go:1199] Started watching for new ooms in manager Nov 04 00:56:20.241443 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.241424 2577 manager.go:319] Starting recovery of all containers Nov 04 00:56:20.253896 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.253870 2577 manager.go:324] Recovery completed Nov 04 00:56:20.259133 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.259121 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:20.261133 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.261118 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:20.261190 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.261143 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:20.261190 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.261153 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:20.261654 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.261639 2577 cpu_manager.go:226] "Starting" policy="none" Nov 04 00:56:20.261654 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.261649 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 00:56:20.261722 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.261665 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 00:56:20.263018 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.263007 2577 policy_none.go:50] "Start" Nov 04 00:56:20.263056 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.263021 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 00:56:20.263056 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.263030 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 00:56:20.263977 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.263967 2577 policy_none.go:44] "Start" Nov 04 00:56:20.264829 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.264816 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:20.265672 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.265644 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:20.265672 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.265669 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:20.265794 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.265680 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:20.309987 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.309959 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 00:56:20.313167 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.313154 2577 manager.go:341] "Starting Device Plugin manager" Nov 04 00:56:20.313252 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:20.313188 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 00:56:20.313252 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.313200 2577 server.go:87] "Starting device plugin registration server" Nov 04 00:56:20.313488 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.313473 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 00:56:20.313550 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.313495 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 00:56:20.313645 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.313618 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 00:56:20.313730 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.313716 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 00:56:20.313730 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.313725 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 00:56:20.314224 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:20.314142 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 00:56:20.314224 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:20.314200 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-94.us-west-2.compute.internal\" not found" Nov 04 00:56:20.376299 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:20.376271 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-94.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 00:56:20.376299 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.376284 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-94.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 00:56:20.377342 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:20.376260 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-94.us-west-2.compute.internal.18e4b18f352a9c17 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-94.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-94.us-west-2.compute.internal,},FirstTimestamp:2026-11-04 00:56:20.230724631 +0000 UTC m=+0.374776366,LastTimestamp:2026-11-04 00:56:20.230724631 +0000 UTC m=+0.374776366,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-94.us-west-2.compute.internal,}" Nov 04 00:56:20.383258 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.383233 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 00:56:20.384414 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.384390 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 00:56:20.384492 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.384421 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 00:56:20.384492 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.384454 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 00:56:20.384577 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:20.384494 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 00:56:20.414007 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.413964 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:20.414805 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.414791 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:20.414875 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.414817 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:20.414875 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.414865 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:20.414957 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.414890 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.444424 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.444404 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 00:56:20.444507 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.444441 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5f59z" Nov 04 00:56:20.485217 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.485194 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal"] Nov 04 00:56:20.485300 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.485262 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:20.486262 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.486244 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:20.486353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.486276 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:20.486353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.486286 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:20.487495 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.487483 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:20.487668 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.487654 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.487703 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.487685 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:20.488270 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.488249 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:20.488270 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.488259 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:20.488397 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.488277 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:20.488397 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.488284 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:20.488397 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.488291 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:20.488397 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.488295 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:20.489405 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.489392 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.489462 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.489417 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:20.490110 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.490095 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:20.490187 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.490119 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:20.490187 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.490128 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:20.517951 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:20.517933 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-94.us-west-2.compute.internal\" not found" node="ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.521931 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:20.521913 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-94.us-west-2.compute.internal\" not found" node="ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.533197 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.533176 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:20.541430 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.541412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7d4196ef8a29ed098777bd3e813be05-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"f7d4196ef8a29ed098777bd3e813be05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.541514 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.541437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7d4196ef8a29ed098777bd3e813be05-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"f7d4196ef8a29ed098777bd3e813be05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.541514 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.541456 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48bdaec858a4ad8070f76d6d06532a70-config\") pod \"kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"48bdaec858a4ad8070f76d6d06532a70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.624684 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.624455 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:20.625066 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.625049 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:20.626265 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.626250 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:20.627262 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.627236 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.627262 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:20.627260 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-94.us-west-2.compute.internal\": node \"ip-10-0-136-94.us-west-2.compute.internal\" not found" Nov 04 00:56:20.639878 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.639864 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.642048 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.642032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48bdaec858a4ad8070f76d6d06532a70-config\") pod \"kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"48bdaec858a4ad8070f76d6d06532a70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.642121 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.642062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7d4196ef8a29ed098777bd3e813be05-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"f7d4196ef8a29ed098777bd3e813be05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.642175 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.642087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7d4196ef8a29ed098777bd3e813be05-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"f7d4196ef8a29ed098777bd3e813be05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.642175 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.642128 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48bdaec858a4ad8070f76d6d06532a70-config\") pod \"kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"48bdaec858a4ad8070f76d6d06532a70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.642175 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.642148 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7d4196ef8a29ed098777bd3e813be05-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"f7d4196ef8a29ed098777bd3e813be05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.642175 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.642148 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7d4196ef8a29ed098777bd3e813be05-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"f7d4196ef8a29ed098777bd3e813be05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.697330 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.697250 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:56:20.698054 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.698040 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.754665 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.754637 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:56:20.821614 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.821583 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:20.825740 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:20.825431 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:56:22.589216 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.589171 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 00:56:22.589729 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.589407 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 00:56:22.589729 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.589409 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 00:56:22.589729 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.589437 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 00:56:22.589729 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:22.589375 2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ab4f6215569114d8cb596808b4941cb0-c1f4c936e9dcd2c2.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-136-94.us-west-2.compute.internal.18e4b18f36fa9c3a\": read tcp 10.0.136.94:45580->3.131.145.49:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-136-94.us-west-2.compute.internal.18e4b18f36fa9c3a default 4902 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-94.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-94.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-94.us-west-2.compute.internal,},FirstTimestamp:2026-11-04 00:56:20 +0000 UTC,LastTimestamp:2026-11-04 00:56:20.486258556 +0000 UTC m=+0.630310289,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-94.us-west-2.compute.internal,}" Nov 04 00:56:22.627097 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.627063 2577 apiserver.go:52] "Watching apiserver" Nov 04 00:56:22.652841 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.652813 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 00:56:22.861620 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.861544 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:51:21 +0000 UTC" deadline="2028-04-27 14:20:32.143472946 +0000 UTC" Nov 04 00:56:22.861620 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.861573 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12973h24m9.281902905s" Nov 04 00:56:22.929244 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.929212 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 00:56:22.929545 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.929524 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq","openshift-cluster-node-tuning-operator/tuned-dr657","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal","openshift-multus/multus-jznmq","openshift-multus/network-metrics-daemon-z5rxm","openshift-network-diagnostics/network-check-target-9slf7","openshift-network-operator/iptables-alerter-ps7mf","openshift-ovn-kubernetes/ovnkube-node-9z6k8","kube-system/global-pull-secret-syncer-pclxc","kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal","openshift-dns/node-resolver-rvxjh","openshift-multus/multus-additional-cni-plugins-wvng4","openshift-image-registry/node-ca-vbxr9","kube-system/konnectivity-agent-7smhm"] Nov 04 00:56:22.930920 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.930906 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:22.930989 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:22.930972 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:22.932012 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.931999 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.933225 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.933201 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:22.934605 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.934587 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.935714 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.935686 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:56:22.936707 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.936687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:22.936791 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:22.936755 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:22.938062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.938045 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:22.939205 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.939190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.940493 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.940479 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:22.940582 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:22.940554 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:22.941739 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.941720 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:56:22.942736 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.942720 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:56:22.943781 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.943764 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:56:22.954301 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.954286 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 00:56:22.961557 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d15630f9-042e-4c31-8f70-c8733c9dc31e-agent-certs\") pod \"konnectivity-agent-7smhm\" (UID: \"d15630f9-042e-4c31-8f70-c8733c9dc31e\") " pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:56:22.961610 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961565 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.961610 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysconfig\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.961610 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:22.961701 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-cnibin\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.961701 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961697 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceb1e5e7-27de-4c20-9576-52d3989384b7-cni-binary-copy\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.961757 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/762f4cf8-0912-4663-94e4-ef0c84740817-host\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:56:22.961757 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961728 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-lib-modules\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.961757 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-registration-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:22.961845 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-ovn\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.961845 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961789 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-kubernetes\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.961845 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-var-lib-kubelet\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.961845 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df3f7f74-c7a1-4f0d-8328-39f4047b6424-tmp\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.961955 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff8l8\" (UniqueName: \"kubernetes.io/projected/df3f7f74-c7a1-4f0d-8328-39f4047b6424-kube-api-access-ff8l8\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.961955 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961892 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:22.961955 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-cni-bin\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.961955 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d15630f9-042e-4c31-8f70-c8733c9dc31e-konnectivity-ca\") pod \"konnectivity-agent-7smhm\" (UID: \"d15630f9-042e-4c31-8f70-c8733c9dc31e\") " pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:56:22.961955 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ebe7f81-60bb-423b-a871-c2bab48db5c3-dbus\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:22.962096 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-system-cni-dir\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:22.962096 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.961995 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:22.962096 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962023 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-cni-multus\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.962096 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-socket-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:22.962096 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-socket-dir-parent\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.962248 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962104 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-daemon-config\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.962248 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:22.962248 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-kubelet\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.962248 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-systemd\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.962248 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-env-overrides\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.962248 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962245 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-system-cni-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.962424 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962266 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkx26\" (UniqueName: \"kubernetes.io/projected/78bd204f-98ed-42b0-9c0d-97baf9aa090d-kube-api-access-fkx26\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:22.962424 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-systemd-units\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.962424 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-slash\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.962424 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovn-node-metrics-cert\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.962424 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-tuned\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.962424 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962421 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cni-binary-copy\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:22.962582 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-k8s-cni-cncf-io\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.962582 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-etc-kubernetes\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.962582 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-cni-netd\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.962582 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962517 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-modprobe-d\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.962582 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962534 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3181e3c4-6f85-4302-965b-672bbf7534ba-iptables-alerter-script\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:56:22.962582 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zkfp\" (UniqueName: \"kubernetes.io/projected/3181e3c4-6f85-4302-965b-672bbf7534ba-kube-api-access-6zkfp\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:56:22.962582 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/762f4cf8-0912-4663-94e4-ef0c84740817-serviceca\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:56:22.962767 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:22.962767 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962612 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cnibin\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:22.962767 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962628 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc65g\" (UniqueName: \"kubernetes.io/projected/2c89163b-5d76-4794-93bb-56840ed333d7-kube-api-access-jc65g\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:22.962767 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962651 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb6fq\" (UniqueName: \"kubernetes.io/projected/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-kube-api-access-mb6fq\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.962767 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/011d3a1a-a2a5-4ef3-a871-453fb22817d5-hosts-file\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:56:22.962767 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-systemd\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.962767 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-sys\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.962767 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-cni-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.963014 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7sjz\" (UniqueName: \"kubernetes.io/projected/ceb1e5e7-27de-4c20-9576-52d3989384b7-kube-api-access-n7sjz\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.963014 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-cni-bin\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.963014 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jljcp\" (UniqueName: \"kubernetes.io/projected/011d3a1a-a2a5-4ef3-a871-453fb22817d5-kube-api-access-jljcp\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:56:22.963014 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-run\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.963014 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962864 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4p4d\" (UniqueName: \"kubernetes.io/projected/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-kube-api-access-h4p4d\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:22.963014 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-conf-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.963014 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47n8\" (UniqueName: \"kubernetes.io/projected/762f4cf8-0912-4663-94e4-ef0c84740817-kube-api-access-n47n8\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:56:22.963014 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ebe7f81-60bb-423b-a871-c2bab48db5c3-kubelet-config\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:22.963014 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-run-netns\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.963014 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-var-lib-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.963014 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.962990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scqt8\" (UniqueName: \"kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8\") pod \"network-check-target-9slf7\" (UID: \"7a4f60af-259e-442f-a17b-911beb60fee5\") " pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:22.963014 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963014 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-log-socket\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.963353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963028 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovnkube-config\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.963353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovnkube-script-lib\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.963353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963096 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:22.963353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysctl-d\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.963353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.963353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysctl-conf\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.963353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-host\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:22.963353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963269 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:22.963353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-device-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:22.963353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-kubelet\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.963353 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963346 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-hostroot\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.963635 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-multus-certs\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.963635 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.963635 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/011d3a1a-a2a5-4ef3-a871-453fb22817d5-tmp-dir\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:56:22.963635 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963438 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-etc-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.963635 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-sys-fs\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:22.963635 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-node-log\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:22.963635 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-netns\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:22.963635 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.963508 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3181e3c4-6f85-4302-965b-672bbf7534ba-host-slash\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:56:22.991249 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.990791 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-66h22\"" Nov 04 00:56:22.991249 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.990817 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 00:56:22.991249 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.990911 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:56:22.991249 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.990968 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 00:56:22.991249 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.990995 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 00:56:22.991249 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.991034 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6gr6q\"" Nov 04 00:56:22.991249 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.991072 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 00:56:22.991249 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.991077 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 00:56:22.991249 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.991150 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 00:56:22.991598 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.991533 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qxmlz\"" Nov 04 00:56:22.991652 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.991621 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 00:56:22.992500 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.992482 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:56:22.992594 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.992504 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ql7fx\"" Nov 04 00:56:22.992594 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.992519 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 00:56:22.992594 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.992543 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 00:56:22.994554 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.994540 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 00:56:22.994854 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.994840 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2w2b9\"" Nov 04 00:56:22.994894 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.994842 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 00:56:22.995322 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.995309 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 00:56:22.995566 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.995554 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 00:56:22.995956 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:22.995945 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4d9rj\"" Nov 04 00:56:23.041293 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.041265 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 00:56:23.041411 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.041291 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 00:56:23.041411 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.041291 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 00:56:23.041411 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.041344 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 00:56:23.041411 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.041400 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gczmn\"" Nov 04 00:56:23.041576 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.041413 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 00:56:23.042534 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.042517 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 00:56:23.042607 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.042538 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 00:56:23.042607 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.042538 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 00:56:23.044611 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.044591 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 00:56:23.044714 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.044706 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 00:56:23.044762 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.044733 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6tf45\"" Nov 04 00:56:23.044762 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.044739 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 00:56:23.045044 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.045029 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 00:56:23.045084 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.045076 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tdv96\"" Nov 04 00:56:23.045217 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.045198 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 00:56:23.045324 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.045307 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p49nz" Nov 04 00:56:23.064181 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-system-cni-dir\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.064283 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.064283 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-cni-multus\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.064283 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064225 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-system-cni-dir\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.064283 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064237 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-socket-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.064283 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064260 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-socket-dir-parent\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.064283 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064271 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-cni-multus\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.064283 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064282 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-daemon-config\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064305 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064330 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-kubelet\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-systemd\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064363 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-socket-dir-parent\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-env-overrides\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064383 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-system-cni-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064423 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-kubelet\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkx26\" (UniqueName: \"kubernetes.io/projected/78bd204f-98ed-42b0-9c0d-97baf9aa090d-kube-api-access-fkx26\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064436 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-systemd\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064479 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-system-cni-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-systemd-units\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-slash\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064579 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-systemd-units\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.064602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovn-node-metrics-cert\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-tuned\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064624 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-slash\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cni-binary-copy\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-k8s-cni-cncf-io\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-etc-kubernetes\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-cni-netd\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-modprobe-d\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064747 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-k8s-cni-cncf-io\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3181e3c4-6f85-4302-965b-672bbf7534ba-iptables-alerter-script\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064778 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-cni-netd\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zkfp\" (UniqueName: \"kubernetes.io/projected/3181e3c4-6f85-4302-965b-672bbf7534ba-kube-api-access-6zkfp\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064804 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-etc-kubernetes\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/762f4cf8-0912-4663-94e4-ef0c84740817-serviceca\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064856 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-modprobe-d\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cnibin\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.065348 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc65g\" (UniqueName: \"kubernetes.io/projected/2c89163b-5d76-4794-93bb-56840ed333d7-kube-api-access-jc65g\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb6fq\" (UniqueName: \"kubernetes.io/projected/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-kube-api-access-mb6fq\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/011d3a1a-a2a5-4ef3-a871-453fb22817d5-hosts-file\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064990 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-socket-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.064992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-systemd\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065006 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-sys\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-cni-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065048 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-systemd\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7sjz\" (UniqueName: \"kubernetes.io/projected/ceb1e5e7-27de-4c20-9576-52d3989384b7-kube-api-access-n7sjz\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-cni-bin\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljcp\" (UniqueName: \"kubernetes.io/projected/011d3a1a-a2a5-4ef3-a871-453fb22817d5-kube-api-access-jljcp\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-run\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.065155 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4p4d\" (UniqueName: \"kubernetes.io/projected/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-kube-api-access-h4p4d\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-conf-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065214 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-cni-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.066111 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.065241 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs podName:78bd204f-98ed-42b0-9c0d-97baf9aa090d nodeName:}" failed. No retries permitted until 2026-11-04 00:56:23.565204075 +0000 UTC m=+2.295452269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs") pod "network-metrics-daemon-z5rxm" (UID: "78bd204f-98ed-42b0-9c0d-97baf9aa090d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065260 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-sys\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065261 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-cni-bin\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065238 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-conf-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065348 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/011d3a1a-a2a5-4ef3-a871-453fb22817d5-hosts-file\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47n8\" (UniqueName: \"kubernetes.io/projected/762f4cf8-0912-4663-94e4-ef0c84740817-kube-api-access-n47n8\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065426 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cnibin\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ebe7f81-60bb-423b-a871-c2bab48db5c3-kubelet-config\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065452 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-daemon-config\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-run-netns\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065495 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/762f4cf8-0912-4663-94e4-ef0c84740817-serviceca\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-var-lib-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065528 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-run\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065558 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-run-netns\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqt8\" (UniqueName: \"kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8\") pod \"network-check-target-9slf7\" (UID: \"7a4f60af-259e-442f-a17b-911beb60fee5\") " pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065568 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ebe7f81-60bb-423b-a871-c2bab48db5c3-kubelet-config\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065593 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-var-lib-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-log-socket\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.066904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065618 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-log-socket\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovnkube-config\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovnkube-script-lib\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysctl-d\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysctl-conf\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-host\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.065791 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065802 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-device-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.065837 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret podName:0ebe7f81-60bb-423b-a871-c2bab48db5c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:23.565821998 +0000 UTC m=+2.296070182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret") pod "global-pull-secret-syncer-pclxc" (UID: "0ebe7f81-60bb-423b-a871-c2bab48db5c3") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065846 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysctl-d\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-kubelet\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065875 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-hostroot\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065908 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-host\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.067775 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-multus-certs\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065936 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-kubelet\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065963 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-hostroot\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065976 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065975 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysctl-conf\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065989 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-multus-certs\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.065881 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-device-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066000 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066000 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/011d3a1a-a2a5-4ef3-a871-453fb22817d5-tmp-dir\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-etc-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-sys-fs\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066088 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-etc-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-node-log\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066158 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-node-log\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-netns\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066204 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-sys-fs\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.068449 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066213 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-netns\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066311 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3181e3c4-6f85-4302-965b-672bbf7534ba-host-slash\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066515 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/011d3a1a-a2a5-4ef3-a871-453fb22817d5-tmp-dir\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066549 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-env-overrides\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066594 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3181e3c4-6f85-4302-965b-672bbf7534ba-iptables-alerter-script\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066635 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cni-binary-copy\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3181e3c4-6f85-4302-965b-672bbf7534ba-host-slash\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d15630f9-042e-4c31-8f70-c8733c9dc31e-agent-certs\") pod \"konnectivity-agent-7smhm\" (UID: \"d15630f9-042e-4c31-8f70-c8733c9dc31e\") " pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066759 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066785 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysconfig\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-cnibin\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceb1e5e7-27de-4c20-9576-52d3989384b7-cni-binary-copy\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066878 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/762f4cf8-0912-4663-94e4-ef0c84740817-host\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066937 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-lib-modules\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066952 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovnkube-script-lib\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.066965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-registration-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.069267 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067003 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-ovn\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067017 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-registration-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067028 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-kubernetes\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067066 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-kubernetes\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067082 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067095 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-ovn\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-var-lib-kubelet\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df3f7f74-c7a1-4f0d-8328-39f4047b6424-tmp\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff8l8\" (UniqueName: \"kubernetes.io/projected/df3f7f74-c7a1-4f0d-8328-39f4047b6424-kube-api-access-ff8l8\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-cni-bin\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067260 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d15630f9-042e-4c31-8f70-c8733c9dc31e-konnectivity-ca\") pod \"konnectivity-agent-7smhm\" (UID: \"d15630f9-042e-4c31-8f70-c8733c9dc31e\") " pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067283 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ebe7f81-60bb-423b-a871-c2bab48db5c3-dbus\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067457 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ebe7f81-60bb-423b-a871-c2bab48db5c3-dbus\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067594 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-cni-bin\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067677 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-lib-modules\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067704 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/762f4cf8-0912-4663-94e4-ef0c84740817-host\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:56:23.070027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067739 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-var-lib-kubelet\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.070842 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067738 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysconfig\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.070842 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067793 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-cnibin\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.070842 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.067911 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceb1e5e7-27de-4c20-9576-52d3989384b7-cni-binary-copy\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.070842 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.068914 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovnkube-config\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.070842 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.068921 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d15630f9-042e-4c31-8f70-c8733c9dc31e-konnectivity-ca\") pod \"konnectivity-agent-7smhm\" (UID: \"d15630f9-042e-4c31-8f70-c8733c9dc31e\") " pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:56:23.070842 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.069273 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.070842 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.070068 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.070842 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.070305 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df3f7f74-c7a1-4f0d-8328-39f4047b6424-tmp\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.070842 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.070340 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-tuned\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.070842 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.070514 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovn-node-metrics-cert\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.071330 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.070915 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d15630f9-042e-4c31-8f70-c8733c9dc31e-agent-certs\") pod \"konnectivity-agent-7smhm\" (UID: \"d15630f9-042e-4c31-8f70-c8733c9dc31e\") " pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:56:23.083277 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.083259 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:56:23.125908 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.125766 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:56:23.125908 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.125800 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:56:23.125908 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.125828 2577 projected.go:196] Error preparing data for projected volume kube-api-access-scqt8 for pod openshift-network-diagnostics/network-check-target-9slf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:23.125908 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.125893 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8 podName:7a4f60af-259e-442f-a17b-911beb60fee5 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:23.625874811 +0000 UTC m=+2.356122999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-scqt8" (UniqueName: "kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8") pod "network-check-target-9slf7" (UID: "7a4f60af-259e-442f-a17b-911beb60fee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:23.128373 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.128343 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb6fq\" (UniqueName: \"kubernetes.io/projected/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-kube-api-access-mb6fq\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.128457 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.128403 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7sjz\" (UniqueName: \"kubernetes.io/projected/ceb1e5e7-27de-4c20-9576-52d3989384b7-kube-api-access-n7sjz\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.129040 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.129019 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkx26\" (UniqueName: \"kubernetes.io/projected/78bd204f-98ed-42b0-9c0d-97baf9aa090d-kube-api-access-fkx26\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:23.129139 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.129072 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc65g\" (UniqueName: \"kubernetes.io/projected/2c89163b-5d76-4794-93bb-56840ed333d7-kube-api-access-jc65g\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.129139 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.129074 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4p4d\" (UniqueName: \"kubernetes.io/projected/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-kube-api-access-h4p4d\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.129357 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.129334 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zkfp\" (UniqueName: \"kubernetes.io/projected/3181e3c4-6f85-4302-965b-672bbf7534ba-kube-api-access-6zkfp\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:56:23.129751 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.129736 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47n8\" (UniqueName: \"kubernetes.io/projected/762f4cf8-0912-4663-94e4-ef0c84740817-kube-api-access-n47n8\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:56:23.129904 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.129886 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff8l8\" (UniqueName: \"kubernetes.io/projected/df3f7f74-c7a1-4f0d-8328-39f4047b6424-kube-api-access-ff8l8\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.130797 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.130780 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljcp\" (UniqueName: \"kubernetes.io/projected/011d3a1a-a2a5-4ef3-a871-453fb22817d5-kube-api-access-jljcp\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:56:23.271417 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.271393 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:56:23.296445 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.296418 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jznmq" Nov 04 00:56:23.297521 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.297482 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:56:23.309346 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.309324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:56:23.313786 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.313768 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:56:23.354703 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.354677 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:23.368865 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.368843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:56:23.380784 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:23.380731 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d4196ef8a29ed098777bd3e813be05.slice/crio-abdcb4619f16a6dd66cf8370e60cfcbd1618f909b34cad8d6dd00a0ea97b6691 WatchSource:0}: Error finding container abdcb4619f16a6dd66cf8370e60cfcbd1618f909b34cad8d6dd00a0ea97b6691: Status 404 returned error can't find the container with id abdcb4619f16a6dd66cf8370e60cfcbd1618f909b34cad8d6dd00a0ea97b6691 Nov 04 00:56:23.381314 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:23.381282 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d69ef80_a3b8_4c02_a636_6b30e21e5b2a.slice/crio-7f72adec94ce451d7d63fab7300af4f04d09c96e9f9eab63f87f971a3b599a95 WatchSource:0}: Error finding container 7f72adec94ce451d7d63fab7300af4f04d09c96e9f9eab63f87f971a3b599a95: Status 404 returned error can't find the container with id 7f72adec94ce451d7d63fab7300af4f04d09c96e9f9eab63f87f971a3b599a95 Nov 04 00:56:23.381733 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.381714 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:56:23.382053 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:23.382029 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bdaec858a4ad8070f76d6d06532a70.slice/crio-6ea2674aeefe26451bfec13e94671d6b93ea1ddb36e50af31affbb530e658065 WatchSource:0}: Error finding container 6ea2674aeefe26451bfec13e94671d6b93ea1ddb36e50af31affbb530e658065: Status 404 returned error can't find the container with id 6ea2674aeefe26451bfec13e94671d6b93ea1ddb36e50af31affbb530e658065 Nov 04 00:56:23.383026 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:23.383003 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011d3a1a_a2a5_4ef3_a871_453fb22817d5.slice/crio-49b06b427edd7f210d0b64acff80f9410d003f962b4e1dc21585105e8b31c7e3 WatchSource:0}: Error finding container 49b06b427edd7f210d0b64acff80f9410d003f962b4e1dc21585105e8b31c7e3: Status 404 returned error can't find the container with id 49b06b427edd7f210d0b64acff80f9410d003f962b4e1dc21585105e8b31c7e3 Nov 04 00:56:23.385514 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.385465 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:56:23.390170 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:23.390141 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762f4cf8_0912_4663_94e4_ef0c84740817.slice/crio-c6cd2148f8da7e3f2640a201e9a468c6b9592a186ac5ec4a477b4332b204f25f WatchSource:0}: Error finding container c6cd2148f8da7e3f2640a201e9a468c6b9592a186ac5ec4a477b4332b204f25f: Status 404 returned error can't find the container with id c6cd2148f8da7e3f2640a201e9a468c6b9592a186ac5ec4a477b4332b204f25f Nov 04 00:56:23.551064 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.551035 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:23.570396 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.570355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:23.570583 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.570478 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:23.570583 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.570496 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:23.570583 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.570529 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs podName:78bd204f-98ed-42b0-9c0d-97baf9aa090d nodeName:}" failed. No retries permitted until 2026-11-04 00:56:24.570514234 +0000 UTC m=+3.300762419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs") pod "network-metrics-daemon-z5rxm" (UID: "78bd204f-98ed-42b0-9c0d-97baf9aa090d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:23.570583 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.570579 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:23.570728 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.570618 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret podName:0ebe7f81-60bb-423b-a871-c2bab48db5c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:24.570608582 +0000 UTC m=+3.300856752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret") pod "global-pull-secret-syncer-pclxc" (UID: "0ebe7f81-60bb-423b-a871-c2bab48db5c3") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:23.671536 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.671456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqt8\" (UniqueName: \"kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8\") pod \"network-check-target-9slf7\" (UID: \"7a4f60af-259e-442f-a17b-911beb60fee5\") " pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:23.672033 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.671568 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:56:23.672033 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.671581 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:56:23.672033 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.671589 2577 projected.go:196] Error preparing data for projected volume kube-api-access-scqt8 for pod openshift-network-diagnostics/network-check-target-9slf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:23.672033 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:23.671635 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8 podName:7a4f60af-259e-442f-a17b-911beb60fee5 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:24.671622208 +0000 UTC m=+3.401870389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-scqt8" (UniqueName: "kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8") pod "network-check-target-9slf7" (UID: "7a4f60af-259e-442f-a17b-911beb60fee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:23.730027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.730000 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:23.802622 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.802569 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vbxr9" event={"ID":"762f4cf8-0912-4663-94e4-ef0c84740817","Type":"ContainerStarted","Data":"c6cd2148f8da7e3f2640a201e9a468c6b9592a186ac5ec4a477b4332b204f25f"} Nov 04 00:56:23.803590 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.803572 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rvxjh" event={"ID":"011d3a1a-a2a5-4ef3-a871-453fb22817d5","Type":"ContainerStarted","Data":"49b06b427edd7f210d0b64acff80f9410d003f962b4e1dc21585105e8b31c7e3"} Nov 04 00:56:23.804665 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.804648 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" event={"ID":"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a","Type":"ContainerStarted","Data":"7f72adec94ce451d7d63fab7300af4f04d09c96e9f9eab63f87f971a3b599a95"} Nov 04 00:56:23.805719 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.805690 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" event={"ID":"48bdaec858a4ad8070f76d6d06532a70","Type":"ContainerStarted","Data":"6ea2674aeefe26451bfec13e94671d6b93ea1ddb36e50af31affbb530e658065"} Nov 04 00:56:23.806650 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.806629 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" event={"ID":"f7d4196ef8a29ed098777bd3e813be05","Type":"ContainerStarted","Data":"abdcb4619f16a6dd66cf8370e60cfcbd1618f909b34cad8d6dd00a0ea97b6691"} Nov 04 00:56:23.965987 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:23.965959 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15630f9_042e_4c31_8f70_c8733c9dc31e.slice/crio-3c454bd79a3e2867138ba42d7a6bd571d65d85986eca640be0d9902327f1ee9e WatchSource:0}: Error finding container 3c454bd79a3e2867138ba42d7a6bd571d65d85986eca640be0d9902327f1ee9e: Status 404 returned error can't find the container with id 3c454bd79a3e2867138ba42d7a6bd571d65d85986eca640be0d9902327f1ee9e Nov 04 00:56:23.980510 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:23.980486 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:24.047311 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.047235 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:51:22 +0000 UTC" deadline="2028-07-07 05:34:46.141573282 +0000 UTC" Nov 04 00:56:24.047311 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.047302 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14668h38m22.09427449s" Nov 04 00:56:24.104790 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:24.104761 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad9dfd7_097e_4491_a3b3_ba3c89a52992.slice/crio-ae166c7a149d538c0a9bbd6f65b27c277ccf8d39ef6e3b16efed217fbc5fb74a WatchSource:0}: Error finding container ae166c7a149d538c0a9bbd6f65b27c277ccf8d39ef6e3b16efed217fbc5fb74a: Status 404 returned error can't find the container with id ae166c7a149d538c0a9bbd6f65b27c277ccf8d39ef6e3b16efed217fbc5fb74a Nov 04 00:56:24.154353 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:24.154328 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb1e5e7_27de_4c20_9576_52d3989384b7.slice/crio-0aa691c57a3767004d22bad031b6714c9202828dbc8ac4b0c4a9136a8bf024b2 WatchSource:0}: Error finding container 0aa691c57a3767004d22bad031b6714c9202828dbc8ac4b0c4a9136a8bf024b2: Status 404 returned error can't find the container with id 0aa691c57a3767004d22bad031b6714c9202828dbc8ac4b0c4a9136a8bf024b2 Nov 04 00:56:24.171771 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:24.171746 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3181e3c4_6f85_4302_965b_672bbf7534ba.slice/crio-9ffc847eeba4c623cb9e5255c1d81b432fad4ff7638531bdd0db0112025965b5 WatchSource:0}: Error finding container 9ffc847eeba4c623cb9e5255c1d81b432fad4ff7638531bdd0db0112025965b5: Status 404 returned error can't find the container with id 9ffc847eeba4c623cb9e5255c1d81b432fad4ff7638531bdd0db0112025965b5 Nov 04 00:56:24.179875 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:24.179853 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3f7f74_c7a1_4f0d_8328_39f4047b6424.slice/crio-9b72af406d6087f8b35723f7c1aa3a85633fc9ffbc74ea12af659104cc34443d WatchSource:0}: Error finding container 9b72af406d6087f8b35723f7c1aa3a85633fc9ffbc74ea12af659104cc34443d: Status 404 returned error can't find the container with id 9b72af406d6087f8b35723f7c1aa3a85633fc9ffbc74ea12af659104cc34443d Nov 04 00:56:24.265005 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:24.264965 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c89163b_5d76_4794_93bb_56840ed333d7.slice/crio-3775001e4a4db27460ffabfcefcfb3ba324e51b0a0a952ae48ea5a258c0e45da WatchSource:0}: Error finding container 3775001e4a4db27460ffabfcefcfb3ba324e51b0a0a952ae48ea5a258c0e45da: Status 404 returned error can't find the container with id 3775001e4a4db27460ffabfcefcfb3ba324e51b0a0a952ae48ea5a258c0e45da Nov 04 00:56:24.578208 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.578107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:24.578208 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.578191 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:24.578439 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:24.578320 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:24.578439 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:24.578389 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret podName:0ebe7f81-60bb-423b-a871-c2bab48db5c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:26.57835824 +0000 UTC m=+5.308606430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret") pod "global-pull-secret-syncer-pclxc" (UID: "0ebe7f81-60bb-423b-a871-c2bab48db5c3") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:24.578549 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:24.578451 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:24.578549 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:24.578480 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs podName:78bd204f-98ed-42b0-9c0d-97baf9aa090d nodeName:}" failed. No retries permitted until 2026-11-04 00:56:26.578470594 +0000 UTC m=+5.308718777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs") pod "network-metrics-daemon-z5rxm" (UID: "78bd204f-98ed-42b0-9c0d-97baf9aa090d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:24.678685 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.678643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqt8\" (UniqueName: \"kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8\") pod \"network-check-target-9slf7\" (UID: \"7a4f60af-259e-442f-a17b-911beb60fee5\") " pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:24.679336 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:24.678835 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:56:24.679336 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:24.678858 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:56:24.679336 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:24.678870 2577 projected.go:196] Error preparing data for projected volume kube-api-access-scqt8 for pod openshift-network-diagnostics/network-check-target-9slf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:24.679336 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:24.678923 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8 podName:7a4f60af-259e-442f-a17b-911beb60fee5 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:26.678907303 +0000 UTC m=+5.409155478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-scqt8" (UniqueName: "kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8") pod "network-check-target-9slf7" (UID: "7a4f60af-259e-442f-a17b-911beb60fee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:24.798972 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.798930 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:24.799173 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:24.799061 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:24.799173 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.799161 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:24.799279 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:24.799246 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:24.799338 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.799321 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:24.799406 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:24.799387 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:24.811410 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.811367 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jznmq" event={"ID":"ceb1e5e7-27de-4c20-9576-52d3989384b7","Type":"ContainerStarted","Data":"0aa691c57a3767004d22bad031b6714c9202828dbc8ac4b0c4a9136a8bf024b2"} Nov 04 00:56:24.812872 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.812846 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvng4" event={"ID":"2ad9dfd7-097e-4491-a3b3-ba3c89a52992","Type":"ContainerStarted","Data":"ae166c7a149d538c0a9bbd6f65b27c277ccf8d39ef6e3b16efed217fbc5fb74a"} Nov 04 00:56:24.814827 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.814802 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7smhm" event={"ID":"d15630f9-042e-4c31-8f70-c8733c9dc31e","Type":"ContainerStarted","Data":"3c454bd79a3e2867138ba42d7a6bd571d65d85986eca640be0d9902327f1ee9e"} Nov 04 00:56:24.817330 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.817304 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" event={"ID":"2c89163b-5d76-4794-93bb-56840ed333d7","Type":"ContainerStarted","Data":"3775001e4a4db27460ffabfcefcfb3ba324e51b0a0a952ae48ea5a258c0e45da"} Nov 04 00:56:24.822549 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.822523 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dr657" event={"ID":"df3f7f74-c7a1-4f0d-8328-39f4047b6424","Type":"ContainerStarted","Data":"9b72af406d6087f8b35723f7c1aa3a85633fc9ffbc74ea12af659104cc34443d"} Nov 04 00:56:24.825218 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:24.825195 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ps7mf" event={"ID":"3181e3c4-6f85-4302-965b-672bbf7534ba","Type":"ContainerStarted","Data":"9ffc847eeba4c623cb9e5255c1d81b432fad4ff7638531bdd0db0112025965b5"} Nov 04 00:56:25.048548 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:25.048444 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:51:22 +0000 UTC" deadline="2028-04-01 17:35:36.749189886 +0000 UTC" Nov 04 00:56:25.048548 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:25.048506 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12352h39m11.700688469s" Nov 04 00:56:26.596055 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:26.596019 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:26.596612 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:26.596087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:26.596612 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:26.596235 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:26.596612 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:26.596319 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret podName:0ebe7f81-60bb-423b-a871-c2bab48db5c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:30.596301946 +0000 UTC m=+9.326550132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret") pod "global-pull-secret-syncer-pclxc" (UID: "0ebe7f81-60bb-423b-a871-c2bab48db5c3") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:26.596612 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:26.596388 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:26.596612 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:26.596420 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs podName:78bd204f-98ed-42b0-9c0d-97baf9aa090d nodeName:}" failed. No retries permitted until 2026-11-04 00:56:30.596409111 +0000 UTC m=+9.326657295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs") pod "network-metrics-daemon-z5rxm" (UID: "78bd204f-98ed-42b0-9c0d-97baf9aa090d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:26.697333 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:26.697297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqt8\" (UniqueName: \"kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8\") pod \"network-check-target-9slf7\" (UID: \"7a4f60af-259e-442f-a17b-911beb60fee5\") " pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:26.697542 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:26.697478 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:56:26.697542 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:26.697496 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:56:26.697542 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:26.697508 2577 projected.go:196] Error preparing data for projected volume kube-api-access-scqt8 for pod openshift-network-diagnostics/network-check-target-9slf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:26.697711 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:26.697566 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8 podName:7a4f60af-259e-442f-a17b-911beb60fee5 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:30.697548805 +0000 UTC m=+9.427796989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-scqt8" (UniqueName: "kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8") pod "network-check-target-9slf7" (UID: "7a4f60af-259e-442f-a17b-911beb60fee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:26.799327 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:26.799292 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:26.799531 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:26.799440 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:26.799531 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:26.799504 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:26.799679 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:26.799578 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:26.799679 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:26.799620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:26.799801 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:26.799681 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:28.799526 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:28.798925 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:28.799526 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:28.799239 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:28.799526 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:28.799278 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:28.799526 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:28.799330 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:28.799526 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:28.799412 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:28.799526 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:28.799443 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:28.844983 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:28.844287 2577 generic.go:356] "Generic (PLEG): container finished" podID="f7d4196ef8a29ed098777bd3e813be05" containerID="14d084ee274713bc6669fe17c4f3e4d88b1c22eefda56065fa0103aa3f47986a" exitCode=0 Nov 04 00:56:28.844983 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:28.844798 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" event={"ID":"f7d4196ef8a29ed098777bd3e813be05","Type":"ContainerDied","Data":"14d084ee274713bc6669fe17c4f3e4d88b1c22eefda56065fa0103aa3f47986a"} Nov 04 00:56:28.850024 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:28.850003 2577 generic.go:356] "Generic (PLEG): container finished" podID="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" containerID="b6e690f3991b135b62923eba6b1f5333501b598d8aaf628443efa31bdb945cdc" exitCode=0 Nov 04 00:56:28.850095 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:28.850074 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvng4" event={"ID":"2ad9dfd7-097e-4491-a3b3-ba3c89a52992","Type":"ContainerDied","Data":"b6e690f3991b135b62923eba6b1f5333501b598d8aaf628443efa31bdb945cdc"} Nov 04 00:56:28.859909 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:28.859885 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" event={"ID":"48bdaec858a4ad8070f76d6d06532a70","Type":"ContainerStarted","Data":"288e545b565fd5c17b551208aba6cab8cc14327e3fde88a95b27c2c029e7005a"} Nov 04 00:56:29.219730 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:29.219445 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" podStartSLOduration=7.219424809 podStartE2EDuration="7.219424809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:29.219254745 +0000 UTC m=+7.949502937" watchObservedRunningTime="2026-11-04 00:56:29.219424809 +0000 UTC m=+7.949673001" Nov 04 00:56:29.867354 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:29.867277 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7smhm" event={"ID":"d15630f9-042e-4c31-8f70-c8733c9dc31e","Type":"ContainerStarted","Data":"7b9a08c3ca7deedc50029d1e33d8a63fa396802fb6a0f0be5c7082dd13b039ea"} Nov 04 00:56:29.872746 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:29.872718 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vbxr9" event={"ID":"762f4cf8-0912-4663-94e4-ef0c84740817","Type":"ContainerStarted","Data":"5fb76b2fd038abf5c527bf8e0ade5589f299da5f6de8a1a75b30c5cb755c9168"} Nov 04 00:56:29.878073 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:29.877973 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" event={"ID":"f7d4196ef8a29ed098777bd3e813be05","Type":"ContainerStarted","Data":"e2289426ba9bce29e31ce78928c58b9b000e218410a76ed5596cab242084edf9"} Nov 04 00:56:30.112034 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:30.111967 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7smhm" podStartSLOduration=3.238353685 podStartE2EDuration="8.11194841s" totalImagesPullingTime="4.873594725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:29.990412507 +0000 UTC m=+8.720660698" watchObservedRunningTime="2026-11-04 00:56:30.11194841 +0000 UTC m=+8.842196597" Nov 04 00:56:30.113003 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:30.112956 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vbxr9" podStartSLOduration=2.210627711 podStartE2EDuration="8.112944907s" totalImagesPullingTime="5.902317196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:30.111941361 +0000 UTC m=+8.842189553" watchObservedRunningTime="2026-11-04 00:56:30.112944907 +0000 UTC m=+8.843193097" Nov 04 00:56:30.197816 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:30.197417 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:56:30.198246 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:30.198201 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:56:30.359657 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:30.359590 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" podStartSLOduration=8.359569004 podStartE2EDuration="8.359569004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:30.237047121 +0000 UTC m=+8.967295314" watchObservedRunningTime="2026-11-04 00:56:30.359569004 +0000 UTC m=+9.089817197" Nov 04 00:56:30.636409 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:30.636301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:30.636409 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:30.636376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:30.636631 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:30.636524 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:30.636631 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:30.636589 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret podName:0ebe7f81-60bb-423b-a871-c2bab48db5c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:38.636570437 +0000 UTC m=+17.366818624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret") pod "global-pull-secret-syncer-pclxc" (UID: "0ebe7f81-60bb-423b-a871-c2bab48db5c3") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:30.636847 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:30.636832 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:30.636908 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:30.636879 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs podName:78bd204f-98ed-42b0-9c0d-97baf9aa090d nodeName:}" failed. No retries permitted until 2026-11-04 00:56:38.636866519 +0000 UTC m=+17.367114703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs") pod "network-metrics-daemon-z5rxm" (UID: "78bd204f-98ed-42b0-9c0d-97baf9aa090d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:30.738184 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:30.738156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqt8\" (UniqueName: \"kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8\") pod \"network-check-target-9slf7\" (UID: \"7a4f60af-259e-442f-a17b-911beb60fee5\") " pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:30.738368 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:30.738340 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:56:30.738442 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:30.738372 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:56:30.738442 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:30.738386 2577 projected.go:196] Error preparing data for projected volume kube-api-access-scqt8 for pod openshift-network-diagnostics/network-check-target-9slf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:30.738565 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:30.738458 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8 podName:7a4f60af-259e-442f-a17b-911beb60fee5 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:38.738440044 +0000 UTC m=+17.468688227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-scqt8" (UniqueName: "kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8") pod "network-check-target-9slf7" (UID: "7a4f60af-259e-442f-a17b-911beb60fee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:30.799028 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:30.798994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:30.799215 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:30.799163 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:30.799276 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:30.799244 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:30.799339 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:30.799319 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:30.799396 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:30.799388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:30.799490 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:30.799472 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:31.884488 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:31.884454 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:56:31.884924 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:31.884522 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" event={"ID":"2c89163b-5d76-4794-93bb-56840ed333d7","Type":"ContainerStarted","Data":"470e768aec93b6bf718293e6d9b25459d3ff1754d5b988405906e92f63f4a794"} Nov 04 00:56:32.799027 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:32.798960 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:32.799175 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:32.799079 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:32.799247 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:32.799230 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:32.799351 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:32.799332 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:32.799389 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:32.799356 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:32.799442 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:32.799426 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:32.890135 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:32.890083 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rvxjh" event={"ID":"011d3a1a-a2a5-4ef3-a871-453fb22817d5","Type":"ContainerStarted","Data":"8ac32cdb99b0ec00868de700b6f3830ceed5f73b4c10cc1765f4ba2d51502699"} Nov 04 00:56:32.892829 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:32.892615 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jznmq" event={"ID":"ceb1e5e7-27de-4c20-9576-52d3989384b7","Type":"ContainerStarted","Data":"4a1585e76f1aaab094e6411380030e07b5fae012a07e9b1e490d7ae61ff315fa"} Nov 04 00:56:32.975307 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:32.975270 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 00:56:33.138110 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:33.138026 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rvxjh" podStartSLOduration=2.6302083720000002 podStartE2EDuration="11.138009641s" totalImagesPullingTime="8.507801269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:33.012736846 +0000 UTC m=+11.742985038" watchObservedRunningTime="2026-11-04 00:56:33.138009641 +0000 UTC m=+11.868257831" Nov 04 00:56:33.735198 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:33.734856 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T00:56:32.97529168Z","UUID":"8156c514-2997-44de-816b-99c0f513bc2b","Handler":null,"Name":"","Endpoint":""} Nov 04 00:56:33.736903 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:33.736872 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 00:56:33.736903 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:33.736906 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 00:56:33.896233 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:33.896189 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" event={"ID":"2c89163b-5d76-4794-93bb-56840ed333d7","Type":"ContainerStarted","Data":"13acfbb0c56d5eddc1210ca7ff7a5d50dcb23152e497a59f2d4c9511deb0316c"} Nov 04 00:56:33.898254 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:33.898224 2577 generic.go:356] "Generic (PLEG): container finished" podID="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" containerID="de9469ad3c5350ca9149c58248c31388d455dc9ff505691bc31055471b4cf7b1" exitCode=0 Nov 04 00:56:33.898392 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:33.898314 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvng4" event={"ID":"2ad9dfd7-097e-4491-a3b3-ba3c89a52992","Type":"ContainerDied","Data":"de9469ad3c5350ca9149c58248c31388d455dc9ff505691bc31055471b4cf7b1"} Nov 04 00:56:34.027152 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:34.027043 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jznmq" podStartSLOduration=5.237124666 podStartE2EDuration="13.027025656s" totalImagesPullingTime="7.78990099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:33.137637419 +0000 UTC m=+11.867885613" watchObservedRunningTime="2026-11-04 00:56:34.027025656 +0000 UTC m=+12.757273852" Nov 04 00:56:34.799158 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:34.799065 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:34.799158 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:34.799097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:34.799158 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:34.799078 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:34.799383 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:34.799227 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:34.799383 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:34.799342 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:34.799460 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:34.799415 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:34.902640 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:34.902608 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dr657" event={"ID":"df3f7f74-c7a1-4f0d-8328-39f4047b6424","Type":"ContainerStarted","Data":"9289ccc55460b2c4495f9909825baee50d675463eeaa161f3fffb9bb502d9ee4"} Nov 04 00:56:34.904667 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:34.904643 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ps7mf" event={"ID":"3181e3c4-6f85-4302-965b-672bbf7534ba","Type":"ContainerStarted","Data":"8078519fd3fc775087f3e4832246c0f6d8bf03b532dc46adb7d41be380a01c27"} Nov 04 00:56:35.027727 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:35.027667 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dr657" podStartSLOduration=3.867161018 podStartE2EDuration="14.027649144s" totalImagesPullingTime="10.160488126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:35.026722769 +0000 UTC m=+13.756970960" watchObservedRunningTime="2026-11-04 00:56:35.027649144 +0000 UTC m=+13.757897337" Nov 04 00:56:35.173199 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:35.173076 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ps7mf" podStartSLOduration=4.805748969 podStartE2EDuration="13.173057907s" totalImagesPullingTime="8.367308938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:35.173000626 +0000 UTC m=+13.903248818" watchObservedRunningTime="2026-11-04 00:56:35.173057907 +0000 UTC m=+13.903306143" Nov 04 00:56:35.908891 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:35.908807 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" event={"ID":"2c89163b-5d76-4794-93bb-56840ed333d7","Type":"ContainerStarted","Data":"e2362f00420e87ce2df10d9951772d03552d7d2281cb43c4ae258d4000236e2c"} Nov 04 00:56:35.911180 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:35.911151 2577 generic.go:356] "Generic (PLEG): container finished" podID="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" containerID="151f3979c52025e853d0042ed81400d4b3e5539a7ffdd86d1314b93953ea8ca7" exitCode=0 Nov 04 00:56:35.911293 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:35.911231 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvng4" event={"ID":"2ad9dfd7-097e-4491-a3b3-ba3c89a52992","Type":"ContainerDied","Data":"151f3979c52025e853d0042ed81400d4b3e5539a7ffdd86d1314b93953ea8ca7"} Nov 04 00:56:36.035787 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:36.035720 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" podStartSLOduration=3.8211015010000002 podStartE2EDuration="14.035696094s" totalImagesPullingTime="10.214594593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:36.03555652 +0000 UTC m=+14.765804713" watchObservedRunningTime="2026-11-04 00:56:36.035696094 +0000 UTC m=+14.765944286" Nov 04 00:56:36.799092 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:36.799061 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:36.799285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:36.799069 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:36.799285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:36.799209 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:36.799285 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:36.799206 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:36.799426 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:36.799325 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:36.799426 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:36.799391 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:37.918257 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:37.917916 2577 generic.go:356] "Generic (PLEG): container finished" podID="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" containerID="ff578754400e935269b50ebea664bcd577f265a756e630ed6ab15e57cb54dd1b" exitCode=0 Nov 04 00:56:37.918257 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:37.917997 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvng4" event={"ID":"2ad9dfd7-097e-4491-a3b3-ba3c89a52992","Type":"ContainerDied","Data":"ff578754400e935269b50ebea664bcd577f265a756e630ed6ab15e57cb54dd1b"} Nov 04 00:56:38.698355 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:38.698317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:38.698554 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:38.698392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:38.698554 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:38.698490 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:38.698669 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:38.698558 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret podName:0ebe7f81-60bb-423b-a871-c2bab48db5c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:54.698538478 +0000 UTC m=+33.428786651 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret") pod "global-pull-secret-syncer-pclxc" (UID: "0ebe7f81-60bb-423b-a871-c2bab48db5c3") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:38.698669 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:38.698494 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:38.698669 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:38.698605 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs podName:78bd204f-98ed-42b0-9c0d-97baf9aa090d nodeName:}" failed. No retries permitted until 2026-11-04 00:56:54.698594647 +0000 UTC m=+33.428842819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs") pod "network-metrics-daemon-z5rxm" (UID: "78bd204f-98ed-42b0-9c0d-97baf9aa090d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:38.799446 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:38.799413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqt8\" (UniqueName: \"kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8\") pod \"network-check-target-9slf7\" (UID: \"7a4f60af-259e-442f-a17b-911beb60fee5\") " pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:38.799446 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:38.799426 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:38.799675 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:38.799476 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:38.799675 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:38.799543 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:56:38.799675 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:38.799563 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:56:38.799675 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:38.799577 2577 projected.go:196] Error preparing data for projected volume kube-api-access-scqt8 for pod openshift-network-diagnostics/network-check-target-9slf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:38.799675 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:38.799584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:38.799675 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:38.799587 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:38.799675 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:38.799625 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8 podName:7a4f60af-259e-442f-a17b-911beb60fee5 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:54.799607893 +0000 UTC m=+33.529856077 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-scqt8" (UniqueName: "kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8") pod "network-check-target-9slf7" (UID: "7a4f60af-259e-442f-a17b-911beb60fee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:38.799897 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:38.799671 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:38.799897 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:38.799762 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:39.924709 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:39.924476 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" event={"ID":"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a","Type":"ContainerStarted","Data":"56f56a12cd5ed713c443c429b5b9a4fb9884ab40a0e45a8a7f6a69cda1210e49"} Nov 04 00:56:39.925136 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:39.924773 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" event={"ID":"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a","Type":"ContainerStarted","Data":"f92062a1044bf16816b467ceb269939022f416eacec3fee37a6191864c74c90d"} Nov 04 00:56:39.925136 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:39.924825 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" event={"ID":"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a","Type":"ContainerStarted","Data":"db3d1dd85ccbd956fe5e34577060b12a785581f4b58f607d81745523e1788212"} Nov 04 00:56:40.799240 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:40.799210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:40.799384 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:40.799318 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:40.799384 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:40.799332 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:40.799384 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:40.799343 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:40.799489 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:40.799406 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:40.799489 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:40.799457 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:40.932097 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:40.932065 2577 generic.go:356] "Generic (PLEG): container finished" podID="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" containerID="e9ef453ab5ba52ef95f915b1492cb8ea249e8a85aa9e6402e4f2888e00011e52" exitCode=0 Nov 04 00:56:40.932542 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:40.932099 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvng4" event={"ID":"2ad9dfd7-097e-4491-a3b3-ba3c89a52992","Type":"ContainerDied","Data":"e9ef453ab5ba52ef95f915b1492cb8ea249e8a85aa9e6402e4f2888e00011e52"} Nov 04 00:56:40.934794 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:40.934774 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" event={"ID":"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a","Type":"ContainerStarted","Data":"3b3cc5218553e2b73c4a34a6b1dbf922ba912d14479a2c42b90adb66b2b519dc"} Nov 04 00:56:40.934872 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:40.934841 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" event={"ID":"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a","Type":"ContainerStarted","Data":"be6cb060f7499e8f8f674c35fad93ff94be7c0b216156a2c7ad293c2cfab037d"} Nov 04 00:56:40.934924 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:40.934891 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" event={"ID":"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a","Type":"ContainerStarted","Data":"e6b62f974e8ba46305ee79416c64e7bffa0f9d27675ddec59dbe9b8d2a36cc44"} Nov 04 00:56:41.940417 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:41.940385 2577 generic.go:356] "Generic (PLEG): container finished" podID="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" containerID="55a3767e770d6ada80fa86d4c0a5d999b212513ee0fc6bb6e613ea343a746607" exitCode=0 Nov 04 00:56:41.940883 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:41.940429 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvng4" event={"ID":"2ad9dfd7-097e-4491-a3b3-ba3c89a52992","Type":"ContainerDied","Data":"55a3767e770d6ada80fa86d4c0a5d999b212513ee0fc6bb6e613ea343a746607"} Nov 04 00:56:42.692203 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:42.692172 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:56:42.692370 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:42.692283 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:56:42.692818 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:42.692799 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:56:42.798584 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:42.798544 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:42.798584 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:42.798579 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:42.798778 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:42.798658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:42.798778 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:42.798665 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:42.798778 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:42.798732 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:42.798896 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:42.798801 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:42.945383 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:42.945299 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvng4" event={"ID":"2ad9dfd7-097e-4491-a3b3-ba3c89a52992","Type":"ContainerStarted","Data":"56b4984bacc2cc1024611eaf88135bda35c70776a73dde37f75512af764c2fc3"} Nov 04 00:56:42.948329 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:42.948308 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" event={"ID":"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a","Type":"ContainerStarted","Data":"936e7b23b479ddb2b95c95a2205d31b0f3d7421c3bfe16e3eab0f374b57d3da8"} Nov 04 00:56:43.072160 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:43.072092 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wvng4" podStartSLOduration=8.907225417 podStartE2EDuration="22.072077999s" totalImagesPullingTime="13.164852582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:43.072064268 +0000 UTC m=+21.802312460" watchObservedRunningTime="2026-11-04 00:56:43.072077999 +0000 UTC m=+21.802326192" Nov 04 00:56:44.799244 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:44.799208 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:44.799828 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:44.799304 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:44.799828 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:44.799329 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:44.799828 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:44.799408 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:44.799828 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:44.799306 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:44.799828 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:44.799497 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:44.956732 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:44.956697 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" event={"ID":"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a","Type":"ContainerStarted","Data":"bfc297efcd01971b2d00f0901c91d89f313c13f95097733c9f5628f941ede05e"} Nov 04 00:56:44.956974 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:44.956958 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:44.957029 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:44.957020 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:44.972387 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:44.972359 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:44.972511 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:44.972464 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:45.251957 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:45.251827 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" podStartSLOduration=6.935051555 podStartE2EDuration="23.251812771s" totalImagesPullingTime="16.316761216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:45.196453318 +0000 UTC m=+23.926701510" watchObservedRunningTime="2026-11-04 00:56:45.251812771 +0000 UTC m=+23.982060962" Nov 04 00:56:45.959245 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:45.959212 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:56:46.798532 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:46.798495 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:46.798689 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:46.798501 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:46.798689 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:46.798624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:46.798689 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:46.798627 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:46.798784 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:46.798698 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:46.798784 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:46.798759 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:46.886134 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:46.886098 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z5rxm"] Nov 04 00:56:46.889743 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:46.889719 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pclxc"] Nov 04 00:56:46.916735 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:46.916711 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9slf7"] Nov 04 00:56:46.961406 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:46.961380 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:46.961406 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:46.961393 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:46.961914 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:46.961395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:46.961914 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:46.961442 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:56:46.961914 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:46.961501 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:46.961914 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:46.961560 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:46.962102 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:46.961981 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:48.799457 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:48.799423 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:48.799795 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:48.799529 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:48.799795 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:48.799545 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:48.799795 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:48.799623 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:48.799795 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:48.799669 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:48.799795 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:48.799746 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:50.082130 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:50.082090 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:56:50.082688 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:50.082329 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:56:50.093253 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:50.093209 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" podUID="6d69ef80-a3b8-4c02-a636-6b30e21e5b2a" containerName="ovnkube-controller" probeResult="failure" output="" Nov 04 00:56:50.102102 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:50.102068 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" podUID="6d69ef80-a3b8-4c02-a636-6b30e21e5b2a" containerName="ovnkube-controller" probeResult="failure" output="" Nov 04 00:56:50.798987 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:50.798952 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:50.799227 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:50.799036 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:50.799227 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:50.799053 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:50.799227 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:50.799178 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:50.799389 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:50.799279 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:50.799389 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:50.799334 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:52.798626 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:52.798578 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:52.799099 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:52.798635 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:52.799099 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:52.798699 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:52.799099 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:52.798719 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pclxc" podUID="0ebe7f81-60bb-423b-a871-c2bab48db5c3" Nov 04 00:56:52.799099 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:52.798776 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5rxm" podUID="78bd204f-98ed-42b0-9c0d-97baf9aa090d" Nov 04 00:56:52.799099 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:52.798823 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9slf7" podUID="7a4f60af-259e-442f-a17b-911beb60fee5" Nov 04 00:56:53.104197 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.104173 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeReady" Nov 04 00:56:53.104355 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.104270 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 00:56:53.218881 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.218843 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bnf5w"] Nov 04 00:56:53.298221 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.298191 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gwvrx"] Nov 04 00:56:53.299062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.299036 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bnf5w" Nov 04 00:56:53.314172 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.314148 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-knfg8"] Nov 04 00:56:53.314297 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.314260 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gwvrx" Nov 04 00:56:53.325729 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.325703 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4"] Nov 04 00:56:53.325843 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.325827 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-knfg8" Nov 04 00:56:53.346690 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.346656 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-s85v2"] Nov 04 00:56:53.346838 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.346768 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:56:53.362220 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.362143 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 04 00:56:53.362430 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.362217 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 04 00:56:53.369672 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.369642 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh"] Nov 04 00:56:53.369830 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.369809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.398930 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.398897 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-767db4f785-j87sp"] Nov 04 00:56:53.399086 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.399020 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:56:53.402380 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.402355 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t95k\" (UniqueName: \"kubernetes.io/projected/81991817-9603-48e1-b25f-0392e58a83b5-kube-api-access-2t95k\") pod \"kas-connection-checker-685895c7c9-knfg8\" (UID: \"81991817-9603-48e1-b25f-0392e58a83b5\") " pod="kube-system/kas-connection-checker-685895c7c9-knfg8" Nov 04 00:56:53.402490 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.402384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/21952e92-3601-4969-adbd-6c6ff9b87074-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:56:53.402490 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.402405 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psm57\" (UniqueName: \"kubernetes.io/projected/593868da-9895-4d98-9da1-4fa815c11674-kube-api-access-psm57\") pod \"kas-connection-checker-685895c7c9-bnf5w\" (UID: \"593868da-9895-4d98-9da1-4fa815c11674\") " pod="kube-system/kas-connection-checker-685895c7c9-bnf5w" Nov 04 00:56:53.402558 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.402493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:56:53.402590 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.402564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7hs\" (UniqueName: \"kubernetes.io/projected/b098c99f-60e8-40bd-a431-fa252a476a0b-kube-api-access-nt7hs\") pod \"kas-connection-checker-685895c7c9-gwvrx\" (UID: \"b098c99f-60e8-40bd-a431-fa252a476a0b\") " pod="kube-system/kas-connection-checker-685895c7c9-gwvrx" Nov 04 00:56:53.402622 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.402600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b9md\" (UniqueName: \"kubernetes.io/projected/21952e92-3601-4969-adbd-6c6ff9b87074-kube-api-access-2b9md\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:56:53.404474 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.404456 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 04 00:56:53.404546 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.404474 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2cfvb\"" Nov 04 00:56:53.423201 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.423169 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc"] Nov 04 00:56:53.423336 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.423278 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.435582 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.435557 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-478s5"] Nov 04 00:56:53.435714 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.435667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc" Nov 04 00:56:53.449822 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.449802 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn"] Nov 04 00:56:53.449927 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.449901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.458469 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.458443 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 00:56:53.458600 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.458519 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 04 00:56:53.458600 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.458538 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:56:53.458713 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.458593 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 04 00:56:53.458902 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.458888 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 00:56:53.458975 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.458962 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 00:56:53.459034 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.458962 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 00:56:53.459094 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.459053 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 00:56:53.459290 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.459264 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 04 00:56:53.474148 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.474108 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww"] Nov 04 00:56:53.474254 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.474244 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:56:53.494558 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.494527 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5768bf545b-g67zq"] Nov 04 00:56:53.494695 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.494644 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:56:53.503280 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7hs\" (UniqueName: \"kubernetes.io/projected/b098c99f-60e8-40bd-a431-fa252a476a0b-kube-api-access-nt7hs\") pod \"kas-connection-checker-685895c7c9-gwvrx\" (UID: \"b098c99f-60e8-40bd-a431-fa252a476a0b\") " pod="kube-system/kas-connection-checker-685895c7c9-gwvrx" Nov 04 00:56:53.503380 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgql6\" (UniqueName: \"kubernetes.io/projected/e2534417-f751-4241-b22d-3ea465dc6a19-kube-api-access-zgql6\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.503380 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68909b0c-2aad-4e3f-ab79-d13d21561db5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.503380 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503323 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68909b0c-2aad-4e3f-ab79-d13d21561db5-service-ca-bundle\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.503505 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503487 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2534417-f751-4241-b22d-3ea465dc6a19-serving-cert\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.503538 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2534417-f751-4241-b22d-3ea465dc6a19-trusted-ca\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.503538 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/576edbd7-09bc-44d8-ba89-12af92909132-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:56:53.503592 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvl2j\" (UniqueName: \"kubernetes.io/projected/68909b0c-2aad-4e3f-ab79-d13d21561db5-kube-api-access-bvl2j\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.503623 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.503656 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b9md\" (UniqueName: \"kubernetes.io/projected/21952e92-3601-4969-adbd-6c6ff9b87074-kube-api-access-2b9md\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:56:53.503656 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503646 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68909b0c-2aad-4e3f-ab79-d13d21561db5-tmp\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.503710 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503690 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74c2k\" (UniqueName: \"kubernetes.io/projected/576edbd7-09bc-44d8-ba89-12af92909132-kube-api-access-74c2k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:56:53.503827 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t95k\" (UniqueName: \"kubernetes.io/projected/81991817-9603-48e1-b25f-0392e58a83b5-kube-api-access-2t95k\") pod \"kas-connection-checker-685895c7c9-knfg8\" (UID: \"81991817-9603-48e1-b25f-0392e58a83b5\") " pod="kube-system/kas-connection-checker-685895c7c9-knfg8" Nov 04 00:56:53.503859 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503836 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/21952e92-3601-4969-adbd-6c6ff9b87074-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:56:53.503859 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-default-certificate\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.503924 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psm57\" (UniqueName: \"kubernetes.io/projected/593868da-9895-4d98-9da1-4fa815c11674-kube-api-access-psm57\") pod \"kas-connection-checker-685895c7c9-bnf5w\" (UID: \"593868da-9895-4d98-9da1-4fa815c11674\") " pod="kube-system/kas-connection-checker-685895c7c9-bnf5w" Nov 04 00:56:53.503924 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503906 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llcg6\" (UniqueName: \"kubernetes.io/projected/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-kube-api-access-llcg6\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.503996 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8s82\" (UniqueName: \"kubernetes.io/projected/2fa177a8-889a-4a1e-9616-7d2f81a55c8b-kube-api-access-g8s82\") pod \"volume-data-source-validator-58c9f5f46d-w9wdc\" (UID: \"2fa177a8-889a-4a1e-9616-7d2f81a55c8b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc" Nov 04 00:56:53.503996 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/68909b0c-2aad-4e3f-ab79-d13d21561db5-snapshots\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.503996 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.503991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-stats-auth\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.504100 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.504014 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.504100 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.504061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68909b0c-2aad-4e3f-ab79-d13d21561db5-serving-cert\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.504100 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.504078 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/576edbd7-09bc-44d8-ba89-12af92909132-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:56:53.504210 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.504179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2534417-f751-4241-b22d-3ea465dc6a19-config\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.504288 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.504269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:56:53.504388 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.504374 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 00:56:53.504473 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.504461 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls podName:21952e92-3601-4969-adbd-6c6ff9b87074 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:54.004448085 +0000 UTC m=+32.734696258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bql4" (UID: "21952e92-3601-4969-adbd-6c6ff9b87074") : secret "cluster-monitoring-operator-tls" not found Nov 04 00:56:53.504565 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.504549 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/21952e92-3601-4969-adbd-6c6ff9b87074-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:56:53.512854 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.512836 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-559546cdcc-tvmhg"] Nov 04 00:56:53.512959 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.512943 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.513747 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.513727 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 00:56:53.513838 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.513750 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 00:56:53.513838 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.513764 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 04 00:56:53.513961 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.513916 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 00:56:53.514064 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.513989 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 04 00:56:53.514064 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514001 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 04 00:56:53.514064 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514003 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:56:53.514064 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514041 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 04 00:56:53.514417 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514381 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5lbwr\"" Nov 04 00:56:53.514417 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514402 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 04 00:56:53.514619 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514459 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 04 00:56:53.514619 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514524 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9rklk\"" Nov 04 00:56:53.514619 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514577 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 04 00:56:53.514843 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514620 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wnwfz\"" Nov 04 00:56:53.514843 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514646 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 04 00:56:53.514843 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514703 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:56:53.514843 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514768 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-b7hfb\"" Nov 04 00:56:53.514843 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514838 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9lbk4\"" Nov 04 00:56:53.515260 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.514870 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 04 00:56:53.515260 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.515012 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 04 00:56:53.515434 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.515420 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 04 00:56:53.531210 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.531192 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 04 00:56:53.531478 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.531464 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 04 00:56:53.531536 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.531490 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:56:53.531536 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.531529 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 04 00:56:53.535971 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.535953 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wkh6k"] Nov 04 00:56:53.536075 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.536062 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.551855 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.551837 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 04 00:56:53.558527 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.558512 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:56:53.558677 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.558662 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 04 00:56:53.566671 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.566652 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6"] Nov 04 00:56:53.566802 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.566784 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:56:53.569721 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.569695 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t95k\" (UniqueName: \"kubernetes.io/projected/81991817-9603-48e1-b25f-0392e58a83b5-kube-api-access-2t95k\") pod \"kas-connection-checker-685895c7c9-knfg8\" (UID: \"81991817-9603-48e1-b25f-0392e58a83b5\") " pod="kube-system/kas-connection-checker-685895c7c9-knfg8" Nov 04 00:56:53.569820 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.569757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psm57\" (UniqueName: \"kubernetes.io/projected/593868da-9895-4d98-9da1-4fa815c11674-kube-api-access-psm57\") pod \"kas-connection-checker-685895c7c9-bnf5w\" (UID: \"593868da-9895-4d98-9da1-4fa815c11674\") " pod="kube-system/kas-connection-checker-685895c7c9-bnf5w" Nov 04 00:56:53.569820 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.569796 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7hs\" (UniqueName: \"kubernetes.io/projected/b098c99f-60e8-40bd-a431-fa252a476a0b-kube-api-access-nt7hs\") pod \"kas-connection-checker-685895c7c9-gwvrx\" (UID: \"b098c99f-60e8-40bd-a431-fa252a476a0b\") " pod="kube-system/kas-connection-checker-685895c7c9-gwvrx" Nov 04 00:56:53.570020 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.570001 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b9md\" (UniqueName: \"kubernetes.io/projected/21952e92-3601-4969-adbd-6c6ff9b87074-kube-api-access-2b9md\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:56:53.583268 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.583235 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zm7jt"] Nov 04 00:56:53.584672 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.584492 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6" Nov 04 00:56:53.605194 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605168 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 04 00:56:53.605333 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605215 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2095b82-50e2-40c5-804e-4f91d84d5e03-config\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:56:53.605333 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605259 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 00:56:53.605333 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llcg6\" (UniqueName: \"kubernetes.io/projected/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-kube-api-access-llcg6\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.605333 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605315 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7clq\" (UniqueName: \"kubernetes.io/projected/f84f08f4-67c2-48b5-b939-ab9269a43bfc-kube-api-access-j7clq\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:56:53.605587 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605400 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 00:56:53.605587 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605441 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w95zk\"" Nov 04 00:56:53.605587 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605441 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5hjbn\"" Nov 04 00:56:53.605587 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605489 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-trusted-ca\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.605587 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605516 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 00:56:53.605587 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-bound-sa-token\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.605587 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpl2n\" (UniqueName: \"kubernetes.io/projected/b2095b82-50e2-40c5-804e-4f91d84d5e03-kube-api-access-cpl2n\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:56:53.605898 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-installation-pull-secrets\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.605898 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68909b0c-2aad-4e3f-ab79-d13d21561db5-serving-cert\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.605898 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qb9d\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-kube-api-access-6qb9d\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.605898 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605799 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-bound-sa-token\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.605898 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac49addd-a050-4d12-bdc0-20345e446a15-ca-trust-extracted\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.606085 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605916 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.606085 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.605995 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dwkfx"] Nov 04 00:56:53.606085 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.606085 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2095b82-50e2-40c5-804e-4f91d84d5e03-serving-cert\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:56:53.606085 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgql6\" (UniqueName: \"kubernetes.io/projected/e2534417-f751-4241-b22d-3ea465dc6a19-kube-api-access-zgql6\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.606326 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68909b0c-2aad-4e3f-ab79-d13d21561db5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.606326 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606151 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:53.606326 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68909b0c-2aad-4e3f-ab79-d13d21561db5-service-ca-bundle\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.606326 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606216 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:56:53.606326 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-image-registry-private-configuration\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.606326 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68909b0c-2aad-4e3f-ab79-d13d21561db5-tmp\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.606326 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74c2k\" (UniqueName: \"kubernetes.io/projected/576edbd7-09bc-44d8-ba89-12af92909132-kube-api-access-74c2k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:56:53.606326 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606303 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-default-certificate\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.606326 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8s82\" (UniqueName: \"kubernetes.io/projected/2fa177a8-889a-4a1e-9616-7d2f81a55c8b-kube-api-access-g8s82\") pod \"volume-data-source-validator-58c9f5f46d-w9wdc\" (UID: \"2fa177a8-889a-4a1e-9616-7d2f81a55c8b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dm7x\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-kube-api-access-4dm7x\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606388 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/68909b0c-2aad-4e3f-ab79-d13d21561db5-snapshots\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-stats-auth\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/576edbd7-09bc-44d8-ba89-12af92909132-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606458 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-image-registry-private-configuration\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2534417-f751-4241-b22d-3ea465dc6a19-config\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606527 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-certificates\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606551 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-trusted-ca\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606572 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-installation-pull-secrets\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.606575 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbd1d155-f232-4679-a70a-ac3911aeb391-ca-trust-extracted\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.606633 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs podName:28aa6fa1-33f8-450e-a1f1-d382e7cf6a47 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:54.106614982 +0000 UTC m=+32.836863162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs") pod "router-default-767db4f785-j87sp" (UID: "28aa6fa1-33f8-450e-a1f1-d382e7cf6a47") : secret "router-metrics-certs-default" not found Nov 04 00:56:53.607285 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.606714 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68909b0c-2aad-4e3f-ab79-d13d21561db5-tmp\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.608457 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.607101 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/68909b0c-2aad-4e3f-ab79-d13d21561db5-snapshots\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.608457 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.607496 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68909b0c-2aad-4e3f-ab79-d13d21561db5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.608457 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.607582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-registry-certificates\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.608457 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.607858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2534417-f751-4241-b22d-3ea465dc6a19-serving-cert\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.608457 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.607905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2534417-f751-4241-b22d-3ea465dc6a19-trusted-ca\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.608457 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.607933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/576edbd7-09bc-44d8-ba89-12af92909132-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:56:53.608457 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.608013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvl2j\" (UniqueName: \"kubernetes.io/projected/68909b0c-2aad-4e3f-ab79-d13d21561db5-kube-api-access-bvl2j\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.608457 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.608047 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2534417-f751-4241-b22d-3ea465dc6a19-config\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.608457 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.608085 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68909b0c-2aad-4e3f-ab79-d13d21561db5-service-ca-bundle\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.608457 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.608090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.608457 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.608228 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle podName:28aa6fa1-33f8-450e-a1f1-d382e7cf6a47 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:54.10821163 +0000 UTC m=+32.838459815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle") pod "router-default-767db4f785-j87sp" (UID: "28aa6fa1-33f8-450e-a1f1-d382e7cf6a47") : configmap references non-existent config key: service-ca.crt Nov 04 00:56:53.609543 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.608685 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68909b0c-2aad-4e3f-ab79-d13d21561db5-serving-cert\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.609543 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.608818 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2534417-f751-4241-b22d-3ea465dc6a19-trusted-ca\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.609543 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.608831 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/576edbd7-09bc-44d8-ba89-12af92909132-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:56:53.609543 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.609478 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-default-certificate\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.609731 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.609580 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/576edbd7-09bc-44d8-ba89-12af92909132-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:56:53.610340 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.610323 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2534417-f751-4241-b22d-3ea465dc6a19-serving-cert\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.610340 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.610332 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-stats-auth\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.623885 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.623866 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 04 00:56:53.624105 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.624093 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-djbdf\"" Nov 04 00:56:53.626750 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.626735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bnf5w" Nov 04 00:56:53.630004 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.629988 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bnf5w"] Nov 04 00:56:53.630055 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630018 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gwvrx"] Nov 04 00:56:53.630055 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630040 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-knfg8"] Nov 04 00:56:53.630130 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630062 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-s85v2"] Nov 04 00:56:53.630130 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630081 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh"] Nov 04 00:56:53.630130 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630101 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dwkfx" Nov 04 00:56:53.630218 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630102 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc"] Nov 04 00:56:53.630218 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630153 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-767db4f785-j87sp"] Nov 04 00:56:53.630218 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630173 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww"] Nov 04 00:56:53.630218 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630192 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-478s5"] Nov 04 00:56:53.630218 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630213 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-559546cdcc-tvmhg"] Nov 04 00:56:53.630414 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630231 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5768bf545b-g67zq"] Nov 04 00:56:53.630414 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630252 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dwkfx"] Nov 04 00:56:53.630414 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630272 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4"] Nov 04 00:56:53.630414 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630293 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn"] Nov 04 00:56:53.630414 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630319 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zm7jt"] Nov 04 00:56:53.630414 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630353 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wkh6k"] Nov 04 00:56:53.630414 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.630375 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6"] Nov 04 00:56:53.647619 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.647598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gwvrx" Nov 04 00:56:53.659058 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.658948 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 00:56:53.659058 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.659051 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 04 00:56:53.659263 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.659240 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 00:56:53.660281 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.660262 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 00:56:53.663406 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.663386 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8chh7\"" Nov 04 00:56:53.663494 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.663486 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 00:56:53.666614 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.666597 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llcg6\" (UniqueName: \"kubernetes.io/projected/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-kube-api-access-llcg6\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:53.668314 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.668296 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 00:56:53.669791 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.669774 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-knfg8" Nov 04 00:56:53.671753 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.671733 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74c2k\" (UniqueName: \"kubernetes.io/projected/576edbd7-09bc-44d8-ba89-12af92909132-kube-api-access-74c2k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:56:53.673678 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.673652 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvl2j\" (UniqueName: \"kubernetes.io/projected/68909b0c-2aad-4e3f-ab79-d13d21561db5-kube-api-access-bvl2j\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.673874 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.673859 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8s82\" (UniqueName: \"kubernetes.io/projected/2fa177a8-889a-4a1e-9616-7d2f81a55c8b-kube-api-access-g8s82\") pod \"volume-data-source-validator-58c9f5f46d-w9wdc\" (UID: \"2fa177a8-889a-4a1e-9616-7d2f81a55c8b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc" Nov 04 00:56:53.674391 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.674372 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgql6\" (UniqueName: \"kubernetes.io/projected/e2534417-f751-4241-b22d-3ea465dc6a19-kube-api-access-zgql6\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.705498 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.705464 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 00:56:53.705642 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.705627 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-g2z88\"" Nov 04 00:56:53.709032 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:56:53.709032 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709033 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-image-registry-private-configuration\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.709257 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-certificates\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.709526 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709507 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-certificates\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.709672 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709653 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:56:53.709724 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-trusted-ca\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.709724 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-installation-pull-secrets\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.709828 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709737 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44f4\" (UniqueName: \"kubernetes.io/projected/3bd0fcf4-1163-444a-95cd-be27096da34d-kube-api-access-q44f4\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:53.709828 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbd1d155-f232-4679-a70a-ac3911aeb391-ca-trust-extracted\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.709828 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq6kt\" (UniqueName: \"kubernetes.io/projected/2248ed8a-1d14-4a43-8e03-7bf83ed903a6-kube-api-access-gq6kt\") pod \"network-check-source-6f78d59f99-cm7m6\" (UID: \"2248ed8a-1d14-4a43-8e03-7bf83ed903a6\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6" Nov 04 00:56:53.709828 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd0fcf4-1163-444a-95cd-be27096da34d-config-volume\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:53.709828 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-registry-certificates\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.710062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:56:53.710062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2095b82-50e2-40c5-804e-4f91d84d5e03-config\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:56:53.710062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7clq\" (UniqueName: \"kubernetes.io/projected/f84f08f4-67c2-48b5-b939-ab9269a43bfc-kube-api-access-j7clq\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:56:53.710062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-trusted-ca\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.710062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-bound-sa-token\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.710062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpl2n\" (UniqueName: \"kubernetes.io/projected/b2095b82-50e2-40c5-804e-4f91d84d5e03-kube-api-access-cpl2n\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:56:53.710062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-installation-pull-secrets\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.710062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd0fcf4-1163-444a-95cd-be27096da34d-tmp-dir\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:53.710062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.709988 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wtnl\" (UniqueName: \"kubernetes.io/projected/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e-kube-api-access-5wtnl\") pod \"ingress-canary-dwkfx\" (UID: \"40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e\") " pod="openshift-ingress-canary/ingress-canary-dwkfx" Nov 04 00:56:53.710062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.710008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qb9d\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-kube-api-access-6qb9d\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.710062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.710025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-bound-sa-token\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.710062 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.710045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac49addd-a050-4d12-bdc0-20345e446a15-ca-trust-extracted\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.710632 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.710069 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.710632 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.710087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.710632 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.710105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2095b82-50e2-40c5-804e-4f91d84d5e03-serving-cert\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:56:53.710632 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.710153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:53.710632 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.710171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:56:53.710632 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.710246 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:56:53.710632 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.710287 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls podName:f84f08f4-67c2-48b5-b939-ab9269a43bfc nodeName:}" failed. No retries permitted until 2026-11-04 00:56:54.210275747 +0000 UTC m=+32.940523916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n4pvn" (UID: "f84f08f4-67c2-48b5-b939-ab9269a43bfc") : secret "samples-operator-tls" not found Nov 04 00:56:53.711039 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.711020 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac49addd-a050-4d12-bdc0-20345e446a15-ca-trust-extracted\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.711233 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.711196 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-trusted-ca\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.711343 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.711313 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:56:53.711343 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.711326 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-559546cdcc-tvmhg: secret "image-registry-tls" not found Nov 04 00:56:53.711490 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.711370 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls podName:ac49addd-a050-4d12-bdc0-20345e446a15 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:54.211355993 +0000 UTC m=+32.941604176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls") pod "image-registry-559546cdcc-tvmhg" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15") : secret "image-registry-tls" not found Nov 04 00:56:53.711490 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.711440 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:56:53.711490 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.711452 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5768bf545b-g67zq: secret "image-registry-tls" not found Nov 04 00:56:53.711490 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.711483 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls podName:cbd1d155-f232-4679-a70a-ac3911aeb391 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:54.211472716 +0000 UTC m=+32.941720899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls") pod "image-registry-5768bf545b-g67zq" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391") : secret "image-registry-tls" not found Nov 04 00:56:53.711793 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.711774 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbd1d155-f232-4679-a70a-ac3911aeb391-ca-trust-extracted\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.711913 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.711894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-image-registry-private-configuration\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.711991 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.711946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b64026c1-aa80-4092-829e-91e5e8164afe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:56:53.711991 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.711971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e-cert\") pod \"ingress-canary-dwkfx\" (UID: \"40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e\") " pod="openshift-ingress-canary/ingress-canary-dwkfx" Nov 04 00:56:53.712265 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.711998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dm7x\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-kube-api-access-4dm7x\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.712547 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.712302 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-registry-certificates\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.712834 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.712809 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2095b82-50e2-40c5-804e-4f91d84d5e03-config\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:56:53.713308 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.713112 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:56:53.713308 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.713233 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-installation-pull-secrets\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.713462 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.713357 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-trusted-ca\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.713918 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.713894 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-image-registry-private-configuration\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.714456 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.714411 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-installation-pull-secrets\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.715158 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.715135 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-image-registry-private-configuration\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.716143 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.716079 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2095b82-50e2-40c5-804e-4f91d84d5e03-serving-cert\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:56:53.729704 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.729684 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:56:53.762996 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.759694 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 00:56:53.762996 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.759803 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 00:56:53.783260 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.783225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:56:53.784345 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.784313 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc" Nov 04 00:56:53.788810 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.788652 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-bound-sa-token\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.788810 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.788679 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpl2n\" (UniqueName: \"kubernetes.io/projected/b2095b82-50e2-40c5-804e-4f91d84d5e03-kube-api-access-cpl2n\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:56:53.788810 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.788711 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-bound-sa-token\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.791472 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.791285 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qb9d\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-kube-api-access-6qb9d\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:53.793475 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.793427 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7clq\" (UniqueName: \"kubernetes.io/projected/f84f08f4-67c2-48b5-b939-ab9269a43bfc-kube-api-access-j7clq\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:56:53.802781 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.802041 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dm7x\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-kube-api-access-4dm7x\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:53.814513 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.813046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44f4\" (UniqueName: \"kubernetes.io/projected/3bd0fcf4-1163-444a-95cd-be27096da34d-kube-api-access-q44f4\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:53.814513 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.813135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6kt\" (UniqueName: \"kubernetes.io/projected/2248ed8a-1d14-4a43-8e03-7bf83ed903a6-kube-api-access-gq6kt\") pod \"network-check-source-6f78d59f99-cm7m6\" (UID: \"2248ed8a-1d14-4a43-8e03-7bf83ed903a6\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6" Nov 04 00:56:53.814513 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.813165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd0fcf4-1163-444a-95cd-be27096da34d-config-volume\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:53.814513 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.813204 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:56:53.814513 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.813269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd0fcf4-1163-444a-95cd-be27096da34d-tmp-dir\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:53.814513 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.813300 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wtnl\" (UniqueName: \"kubernetes.io/projected/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e-kube-api-access-5wtnl\") pod \"ingress-canary-dwkfx\" (UID: \"40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e\") " pod="openshift-ingress-canary/ingress-canary-dwkfx" Nov 04 00:56:53.814513 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.813413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:53.814513 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.813462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b64026c1-aa80-4092-829e-91e5e8164afe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:56:53.814513 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.813486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e-cert\") pod \"ingress-canary-dwkfx\" (UID: \"40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e\") " pod="openshift-ingress-canary/ingress-canary-dwkfx" Nov 04 00:56:53.814513 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.814014 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 00:56:53.814513 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.814081 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert podName:b64026c1-aa80-4092-829e-91e5e8164afe nodeName:}" failed. No retries permitted until 2026-11-04 00:56:54.314062877 +0000 UTC m=+33.044311052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wkh6k" (UID: "b64026c1-aa80-4092-829e-91e5e8164afe") : secret "networking-console-plugin-cert" not found Nov 04 00:56:53.815037 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.814725 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd0fcf4-1163-444a-95cd-be27096da34d-tmp-dir\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:53.815037 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.814823 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:56:53.815037 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.814846 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd0fcf4-1163-444a-95cd-be27096da34d-config-volume\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:53.815037 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:53.814875 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls podName:3bd0fcf4-1163-444a-95cd-be27096da34d nodeName:}" failed. No retries permitted until 2026-11-04 00:56:54.314855664 +0000 UTC m=+33.045103847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls") pod "dns-default-zm7jt" (UID: "3bd0fcf4-1163-444a-95cd-be27096da34d") : secret "dns-default-metrics-tls" not found Nov 04 00:56:53.815488 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.815461 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b64026c1-aa80-4092-829e-91e5e8164afe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:56:53.819447 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.819411 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e-cert\") pod \"ingress-canary-dwkfx\" (UID: \"40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e\") " pod="openshift-ingress-canary/ingress-canary-dwkfx" Nov 04 00:56:53.836375 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.836320 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:56:53.877268 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.877190 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wtnl\" (UniqueName: \"kubernetes.io/projected/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e-kube-api-access-5wtnl\") pod \"ingress-canary-dwkfx\" (UID: \"40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e\") " pod="openshift-ingress-canary/ingress-canary-dwkfx" Nov 04 00:56:53.879214 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.879192 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6kt\" (UniqueName: \"kubernetes.io/projected/2248ed8a-1d14-4a43-8e03-7bf83ed903a6-kube-api-access-gq6kt\") pod \"network-check-source-6f78d59f99-cm7m6\" (UID: \"2248ed8a-1d14-4a43-8e03-7bf83ed903a6\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6" Nov 04 00:56:53.879763 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.879743 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44f4\" (UniqueName: \"kubernetes.io/projected/3bd0fcf4-1163-444a-95cd-be27096da34d-kube-api-access-q44f4\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:53.939664 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:53.939633 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6" Nov 04 00:56:54.016845 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.016805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:56:54.016998 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.016975 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 00:56:54.017199 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.017067 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls podName:21952e92-3601-4969-adbd-6c6ff9b87074 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:55.017046484 +0000 UTC m=+33.747294667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bql4" (UID: "21952e92-3601-4969-adbd-6c6ff9b87074") : secret "cluster-monitoring-operator-tls" not found Nov 04 00:56:54.022651 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.022629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dwkfx" Nov 04 00:56:54.108148 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.102769 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bnf5w"] Nov 04 00:56:54.108148 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.105016 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gwvrx"] Nov 04 00:56:54.111826 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:54.111796 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593868da_9895_4d98_9da1_4fa815c11674.slice/crio-fc001fd8e0377241c1f20264c489dbba015c928138f744f48deb41a1accad95c WatchSource:0}: Error finding container fc001fd8e0377241c1f20264c489dbba015c928138f744f48deb41a1accad95c: Status 404 returned error can't find the container with id fc001fd8e0377241c1f20264c489dbba015c928138f744f48deb41a1accad95c Nov 04 00:56:54.112239 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:54.112218 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb098c99f_60e8_40bd_a431_fa252a476a0b.slice/crio-2fd08dc37586bbaccc3bc48aeaf69b34346ad767e38bae7882ea89edbe1f7563 WatchSource:0}: Error finding container 2fd08dc37586bbaccc3bc48aeaf69b34346ad767e38bae7882ea89edbe1f7563: Status 404 returned error can't find the container with id 2fd08dc37586bbaccc3bc48aeaf69b34346ad767e38bae7882ea89edbe1f7563 Nov 04 00:56:54.118805 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.118367 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:54.118805 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.118530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:54.118805 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.118649 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 00:56:54.118805 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.118704 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs podName:28aa6fa1-33f8-450e-a1f1-d382e7cf6a47 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:55.118687409 +0000 UTC m=+33.848935602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs") pod "router-default-767db4f785-j87sp" (UID: "28aa6fa1-33f8-450e-a1f1-d382e7cf6a47") : secret "router-metrics-certs-default" not found Nov 04 00:56:54.118805 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.118778 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle podName:28aa6fa1-33f8-450e-a1f1-d382e7cf6a47 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:55.118768695 +0000 UTC m=+33.849016879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle") pod "router-default-767db4f785-j87sp" (UID: "28aa6fa1-33f8-450e-a1f1-d382e7cf6a47") : configmap references non-existent config key: service-ca.crt Nov 04 00:56:54.153893 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.153792 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-s85v2"] Nov 04 00:56:54.165613 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:54.165579 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68909b0c_2aad_4e3f_ab79_d13d21561db5.slice/crio-6239dd443759994df4c3cd47ab57a764dba24aed318f6e9c06b20f08f7a90ebb WatchSource:0}: Error finding container 6239dd443759994df4c3cd47ab57a764dba24aed318f6e9c06b20f08f7a90ebb: Status 404 returned error can't find the container with id 6239dd443759994df4c3cd47ab57a764dba24aed318f6e9c06b20f08f7a90ebb Nov 04 00:56:54.184637 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.184232 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh"] Nov 04 00:56:54.186809 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:54.186772 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576edbd7_09bc_44d8_ba89_12af92909132.slice/crio-94b25173f5fcdf9544a823009ff9682def60a2f6ad23e1e41e2bdb9644325959 WatchSource:0}: Error finding container 94b25173f5fcdf9544a823009ff9682def60a2f6ad23e1e41e2bdb9644325959: Status 404 returned error can't find the container with id 94b25173f5fcdf9544a823009ff9682def60a2f6ad23e1e41e2bdb9644325959 Nov 04 00:56:54.204595 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.204538 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-478s5"] Nov 04 00:56:54.208104 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:54.208070 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2534417_f751_4241_b22d_3ea465dc6a19.slice/crio-3183dfa600ae4dc32d285f2fa478fba8384bc67ec4e8397aace58f7e1f90a6d3 WatchSource:0}: Error finding container 3183dfa600ae4dc32d285f2fa478fba8384bc67ec4e8397aace58f7e1f90a6d3: Status 404 returned error can't find the container with id 3183dfa600ae4dc32d285f2fa478fba8384bc67ec4e8397aace58f7e1f90a6d3 Nov 04 00:56:54.220429 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.220362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:54.220429 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.220399 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:54.220644 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.220461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:56:54.220644 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.220542 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:56:54.220644 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.220557 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-559546cdcc-tvmhg: secret "image-registry-tls" not found Nov 04 00:56:54.220968 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.220678 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:56:54.220968 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.220680 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls podName:ac49addd-a050-4d12-bdc0-20345e446a15 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:55.220660314 +0000 UTC m=+33.950908505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls") pod "image-registry-559546cdcc-tvmhg" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15") : secret "image-registry-tls" not found Nov 04 00:56:54.220968 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.220738 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls podName:f84f08f4-67c2-48b5-b939-ab9269a43bfc nodeName:}" failed. No retries permitted until 2026-11-04 00:56:55.220726522 +0000 UTC m=+33.950974701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n4pvn" (UID: "f84f08f4-67c2-48b5-b939-ab9269a43bfc") : secret "samples-operator-tls" not found Nov 04 00:56:54.221701 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.221262 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:56:54.221701 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.221282 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5768bf545b-g67zq: secret "image-registry-tls" not found Nov 04 00:56:54.221701 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.221335 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls podName:cbd1d155-f232-4679-a70a-ac3911aeb391 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:55.221319318 +0000 UTC m=+33.951567502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls") pod "image-registry-5768bf545b-g67zq" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391") : secret "image-registry-tls" not found Nov 04 00:56:54.222998 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.222951 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-knfg8"] Nov 04 00:56:54.225935 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:54.225914 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81991817_9603_48e1_b25f_0392e58a83b5.slice/crio-4f37ebc5fcf7f274bdf12d6e3b5ea789795044ce760ffa663c6f5f5a54e681ff WatchSource:0}: Error finding container 4f37ebc5fcf7f274bdf12d6e3b5ea789795044ce760ffa663c6f5f5a54e681ff: Status 404 returned error can't find the container with id 4f37ebc5fcf7f274bdf12d6e3b5ea789795044ce760ffa663c6f5f5a54e681ff Nov 04 00:56:54.230564 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.230546 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc"] Nov 04 00:56:54.234102 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:54.234079 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa177a8_889a_4a1e_9616_7d2f81a55c8b.slice/crio-77ec6df6a03a3682953ac1001b9b484abf1f9f75d62f158da113f8245237946f WatchSource:0}: Error finding container 77ec6df6a03a3682953ac1001b9b484abf1f9f75d62f158da113f8245237946f: Status 404 returned error can't find the container with id 77ec6df6a03a3682953ac1001b9b484abf1f9f75d62f158da113f8245237946f Nov 04 00:56:54.252281 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.252251 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww"] Nov 04 00:56:54.257387 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:54.257359 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2095b82_50e2_40c5_804e_4f91d84d5e03.slice/crio-aa76b32e7ca58dbeb41e98e9dc715191e66fa15a184fc75e80230e779914d124 WatchSource:0}: Error finding container aa76b32e7ca58dbeb41e98e9dc715191e66fa15a184fc75e80230e779914d124: Status 404 returned error can't find the container with id aa76b32e7ca58dbeb41e98e9dc715191e66fa15a184fc75e80230e779914d124 Nov 04 00:56:54.321075 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.321047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:56:54.321245 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.321226 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 00:56:54.321324 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.321236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:54.321324 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.321301 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:56:54.321400 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.321301 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert podName:b64026c1-aa80-4092-829e-91e5e8164afe nodeName:}" failed. No retries permitted until 2026-11-04 00:56:55.321279339 +0000 UTC m=+34.051527508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wkh6k" (UID: "b64026c1-aa80-4092-829e-91e5e8164afe") : secret "networking-console-plugin-cert" not found Nov 04 00:56:54.321400 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.321377 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls podName:3bd0fcf4-1163-444a-95cd-be27096da34d nodeName:}" failed. No retries permitted until 2026-11-04 00:56:55.321355016 +0000 UTC m=+34.051603188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls") pod "dns-default-zm7jt" (UID: "3bd0fcf4-1163-444a-95cd-be27096da34d") : secret "dns-default-metrics-tls" not found Nov 04 00:56:54.338602 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.338579 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6"] Nov 04 00:56:54.341751 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:54.341727 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2248ed8a_1d14_4a43_8e03_7bf83ed903a6.slice/crio-ac1b456d6d1432d94931f8726a3c15cdbde7f3f8e2c8199bb875232786d24fe6 WatchSource:0}: Error finding container ac1b456d6d1432d94931f8726a3c15cdbde7f3f8e2c8199bb875232786d24fe6: Status 404 returned error can't find the container with id ac1b456d6d1432d94931f8726a3c15cdbde7f3f8e2c8199bb875232786d24fe6 Nov 04 00:56:54.417232 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.417150 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dwkfx"] Nov 04 00:56:54.420724 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:54.420691 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40eb6c9a_7483_482b_aaa8_1cc2ca0bb93e.slice/crio-92fe1d699adb2132995a76d9f46052f682f1321c3b82552bcfae33f4bebc2886 WatchSource:0}: Error finding container 92fe1d699adb2132995a76d9f46052f682f1321c3b82552bcfae33f4bebc2886: Status 404 returned error can't find the container with id 92fe1d699adb2132995a76d9f46052f682f1321c3b82552bcfae33f4bebc2886 Nov 04 00:56:54.724194 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.724102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:54.724194 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.724182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:54.724372 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.724247 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:54.724372 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.724304 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs podName:78bd204f-98ed-42b0-9c0d-97baf9aa090d nodeName:}" failed. No retries permitted until 2026-11-04 00:57:26.724291517 +0000 UTC m=+65.454539689 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs") pod "network-metrics-daemon-z5rxm" (UID: "78bd204f-98ed-42b0-9c0d-97baf9aa090d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:54.724372 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.724322 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:54.724372 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:54.724366 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret podName:0ebe7f81-60bb-423b-a871-c2bab48db5c3 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:26.724355166 +0000 UTC m=+65.454603363 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret") pod "global-pull-secret-syncer-pclxc" (UID: "0ebe7f81-60bb-423b-a871-c2bab48db5c3") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:54.798920 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.798879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:56:54.799077 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.798937 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:54.799077 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.799035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:56:54.824900 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.824878 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqt8\" (UniqueName: \"kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8\") pod \"network-check-target-9slf7\" (UID: \"7a4f60af-259e-442f-a17b-911beb60fee5\") " pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:54.827311 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.827292 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scqt8\" (UniqueName: \"kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8\") pod \"network-check-target-9slf7\" (UID: \"7a4f60af-259e-442f-a17b-911beb60fee5\") " pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:54.857237 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.857202 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-47hrz\"" Nov 04 00:56:54.857237 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.857212 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 00:56:54.857366 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.857277 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fh9wk\"" Nov 04 00:56:54.857366 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.857281 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 00:56:54.857366 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.857300 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xq5nc\"" Nov 04 00:56:54.916693 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.916668 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:56:54.983402 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.983368 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" event={"ID":"576edbd7-09bc-44d8-ba89-12af92909132","Type":"ContainerStarted","Data":"94b25173f5fcdf9544a823009ff9682def60a2f6ad23e1e41e2bdb9644325959"} Nov 04 00:56:54.986597 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.986571 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6" event={"ID":"2248ed8a-1d14-4a43-8e03-7bf83ed903a6","Type":"ContainerStarted","Data":"ac1b456d6d1432d94931f8726a3c15cdbde7f3f8e2c8199bb875232786d24fe6"} Nov 04 00:56:54.987972 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.987945 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-knfg8" event={"ID":"81991817-9603-48e1-b25f-0392e58a83b5","Type":"ContainerStarted","Data":"25cce53682b86571564f1df2cde70b01f6a6dae99d66b96e6328c6221ce656cc"} Nov 04 00:56:54.988066 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.987997 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-knfg8" event={"ID":"81991817-9603-48e1-b25f-0392e58a83b5","Type":"ContainerStarted","Data":"4f37ebc5fcf7f274bdf12d6e3b5ea789795044ce760ffa663c6f5f5a54e681ff"} Nov 04 00:56:54.989380 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.989360 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bnf5w" event={"ID":"593868da-9895-4d98-9da1-4fa815c11674","Type":"ContainerStarted","Data":"e0e56377b7751166d2fd7f1b1e99a17b71e0feacab6eda40c6aa0f13479440bd"} Nov 04 00:56:54.989445 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.989401 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bnf5w" event={"ID":"593868da-9895-4d98-9da1-4fa815c11674","Type":"ContainerStarted","Data":"fc001fd8e0377241c1f20264c489dbba015c928138f744f48deb41a1accad95c"} Nov 04 00:56:54.990476 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.990450 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc" event={"ID":"2fa177a8-889a-4a1e-9616-7d2f81a55c8b","Type":"ContainerStarted","Data":"77ec6df6a03a3682953ac1001b9b484abf1f9f75d62f158da113f8245237946f"} Nov 04 00:56:54.991878 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.991853 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gwvrx" event={"ID":"b098c99f-60e8-40bd-a431-fa252a476a0b","Type":"ContainerStarted","Data":"09012f096e265b62117ce783979569eab16488f9262857cf95518d5d37e70a6f"} Nov 04 00:56:54.991966 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.991892 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gwvrx" event={"ID":"b098c99f-60e8-40bd-a431-fa252a476a0b","Type":"ContainerStarted","Data":"2fd08dc37586bbaccc3bc48aeaf69b34346ad767e38bae7882ea89edbe1f7563"} Nov 04 00:56:54.992921 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.992898 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" event={"ID":"e2534417-f751-4241-b22d-3ea465dc6a19","Type":"ContainerStarted","Data":"3183dfa600ae4dc32d285f2fa478fba8384bc67ec4e8397aace58f7e1f90a6d3"} Nov 04 00:56:54.994379 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.994333 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-s85v2" event={"ID":"68909b0c-2aad-4e3f-ab79-d13d21561db5","Type":"ContainerStarted","Data":"6239dd443759994df4c3cd47ab57a764dba24aed318f6e9c06b20f08f7a90ebb"} Nov 04 00:56:54.995681 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.995660 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dwkfx" event={"ID":"40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e","Type":"ContainerStarted","Data":"92fe1d699adb2132995a76d9f46052f682f1321c3b82552bcfae33f4bebc2886"} Nov 04 00:56:54.996897 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:54.996863 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" event={"ID":"b2095b82-50e2-40c5-804e-4f91d84d5e03","Type":"ContainerStarted","Data":"aa76b32e7ca58dbeb41e98e9dc715191e66fa15a184fc75e80230e779914d124"} Nov 04 00:56:55.026881 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:55.026848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:56:55.027005 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.026936 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 00:56:55.027005 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.027001 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls podName:21952e92-3601-4969-adbd-6c6ff9b87074 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:57.026984305 +0000 UTC m=+35.757232490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bql4" (UID: "21952e92-3601-4969-adbd-6c6ff9b87074") : secret "cluster-monitoring-operator-tls" not found Nov 04 00:56:55.128157 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:55.128107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:55.128307 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:55.128213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:55.128307 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.128254 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 00:56:55.128371 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.128321 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs podName:28aa6fa1-33f8-450e-a1f1-d382e7cf6a47 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:57.128303249 +0000 UTC m=+35.858551438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs") pod "router-default-767db4f785-j87sp" (UID: "28aa6fa1-33f8-450e-a1f1-d382e7cf6a47") : secret "router-metrics-certs-default" not found Nov 04 00:56:55.128409 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.128383 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle podName:28aa6fa1-33f8-450e-a1f1-d382e7cf6a47 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:57.128373836 +0000 UTC m=+35.858622005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle") pod "router-default-767db4f785-j87sp" (UID: "28aa6fa1-33f8-450e-a1f1-d382e7cf6a47") : configmap references non-existent config key: service-ca.crt Nov 04 00:56:55.229294 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:55.229225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:55.229294 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:55.229256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:55.229495 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.229367 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:56:55.229495 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.229379 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5768bf545b-g67zq: secret "image-registry-tls" not found Nov 04 00:56:55.229495 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:55.229405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:56:55.229495 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.229421 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls podName:cbd1d155-f232-4679-a70a-ac3911aeb391 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:57.229409613 +0000 UTC m=+35.959657801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls") pod "image-registry-5768bf545b-g67zq" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391") : secret "image-registry-tls" not found Nov 04 00:56:55.229495 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.229367 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:56:55.229495 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.229437 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-559546cdcc-tvmhg: secret "image-registry-tls" not found Nov 04 00:56:55.229495 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.229463 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls podName:ac49addd-a050-4d12-bdc0-20345e446a15 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:57.229457442 +0000 UTC m=+35.959705611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls") pod "image-registry-559546cdcc-tvmhg" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15") : secret "image-registry-tls" not found Nov 04 00:56:55.229495 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.229493 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:56:55.229739 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.229541 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls podName:f84f08f4-67c2-48b5-b939-ab9269a43bfc nodeName:}" failed. No retries permitted until 2026-11-04 00:56:57.229528976 +0000 UTC m=+35.959777158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n4pvn" (UID: "f84f08f4-67c2-48b5-b939-ab9269a43bfc") : secret "samples-operator-tls" not found Nov 04 00:56:55.300247 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:55.300207 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9slf7"] Nov 04 00:56:55.303399 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:56:55.303158 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a4f60af_259e_442f_a17b_911beb60fee5.slice/crio-2a2d786d3267c9a6dfe3c4e55d983cc405632b2f97a5ebcbb6188ca0af23333a WatchSource:0}: Error finding container 2a2d786d3267c9a6dfe3c4e55d983cc405632b2f97a5ebcbb6188ca0af23333a: Status 404 returned error can't find the container with id 2a2d786d3267c9a6dfe3c4e55d983cc405632b2f97a5ebcbb6188ca0af23333a Nov 04 00:56:55.330416 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:55.330380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:55.330574 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:55.330497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:56:55.330574 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.330527 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:56:55.330676 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.330582 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls podName:3bd0fcf4-1163-444a-95cd-be27096da34d nodeName:}" failed. No retries permitted until 2026-11-04 00:56:57.33056504 +0000 UTC m=+36.060813220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls") pod "dns-default-zm7jt" (UID: "3bd0fcf4-1163-444a-95cd-be27096da34d") : secret "dns-default-metrics-tls" not found Nov 04 00:56:55.330676 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.330637 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 00:56:55.330750 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:55.330681 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert podName:b64026c1-aa80-4092-829e-91e5e8164afe nodeName:}" failed. No retries permitted until 2026-11-04 00:56:57.330672911 +0000 UTC m=+36.060921084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wkh6k" (UID: "b64026c1-aa80-4092-829e-91e5e8164afe") : secret "networking-console-plugin-cert" not found Nov 04 00:56:55.577264 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:55.576963 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-knfg8" podStartSLOduration=457.576944481 podStartE2EDuration="7m37.576944481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:55.576170772 +0000 UTC m=+34.306418963" watchObservedRunningTime="2026-11-04 00:56:55.576944481 +0000 UTC m=+34.307192670" Nov 04 00:56:55.833104 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:55.831748 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-gwvrx" podStartSLOduration=457.831728629 podStartE2EDuration="7m37.831728629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:55.708225665 +0000 UTC m=+34.438473859" watchObservedRunningTime="2026-11-04 00:56:55.831728629 +0000 UTC m=+34.561976821" Nov 04 00:56:56.003329 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:56.002678 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9slf7" event={"ID":"7a4f60af-259e-442f-a17b-911beb60fee5","Type":"ContainerStarted","Data":"2a2d786d3267c9a6dfe3c4e55d983cc405632b2f97a5ebcbb6188ca0af23333a"} Nov 04 00:56:57.049966 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:57.049379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:56:57.049966 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.049569 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 00:56:57.049966 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.049639 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls podName:21952e92-3601-4969-adbd-6c6ff9b87074 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:01.049625343 +0000 UTC m=+39.779873511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bql4" (UID: "21952e92-3601-4969-adbd-6c6ff9b87074") : secret "cluster-monitoring-operator-tls" not found Nov 04 00:56:57.151923 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:57.151858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:57.152578 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.152490 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 00:56:57.152810 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.152684 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs podName:28aa6fa1-33f8-450e-a1f1-d382e7cf6a47 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:01.152658814 +0000 UTC m=+39.882907007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs") pod "router-default-767db4f785-j87sp" (UID: "28aa6fa1-33f8-450e-a1f1-d382e7cf6a47") : secret "router-metrics-certs-default" not found Nov 04 00:56:57.153256 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:57.153034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:56:57.153256 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.153233 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle podName:28aa6fa1-33f8-450e-a1f1-d382e7cf6a47 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:01.153217486 +0000 UTC m=+39.883465669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle") pod "router-default-767db4f785-j87sp" (UID: "28aa6fa1-33f8-450e-a1f1-d382e7cf6a47") : configmap references non-existent config key: service-ca.crt Nov 04 00:56:57.255084 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:57.254717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:56:57.255084 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:57.254768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:56:57.255084 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:57.254822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:56:57.255084 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.254863 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:56:57.255084 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.254880 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-559546cdcc-tvmhg: secret "image-registry-tls" not found Nov 04 00:56:57.255084 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.254939 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls podName:ac49addd-a050-4d12-bdc0-20345e446a15 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:01.254921811 +0000 UTC m=+39.985170000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls") pod "image-registry-559546cdcc-tvmhg" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15") : secret "image-registry-tls" not found Nov 04 00:56:57.255084 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.254982 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:56:57.255084 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.255021 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls podName:f84f08f4-67c2-48b5-b939-ab9269a43bfc nodeName:}" failed. No retries permitted until 2026-11-04 00:57:01.255009368 +0000 UTC m=+39.985257551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n4pvn" (UID: "f84f08f4-67c2-48b5-b939-ab9269a43bfc") : secret "samples-operator-tls" not found Nov 04 00:56:57.255084 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.255062 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:56:57.255084 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.255073 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5768bf545b-g67zq: secret "image-registry-tls" not found Nov 04 00:56:57.255084 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.255100 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls podName:cbd1d155-f232-4679-a70a-ac3911aeb391 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:01.255091377 +0000 UTC m=+39.985339548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls") pod "image-registry-5768bf545b-g67zq" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391") : secret "image-registry-tls" not found Nov 04 00:56:57.356906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:57.356382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:56:57.356906 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:57.356534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:56:57.356906 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.356687 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:56:57.356906 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.356748 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls podName:3bd0fcf4-1163-444a-95cd-be27096da34d nodeName:}" failed. No retries permitted until 2026-11-04 00:57:01.356729425 +0000 UTC m=+40.086977611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls") pod "dns-default-zm7jt" (UID: "3bd0fcf4-1163-444a-95cd-be27096da34d") : secret "dns-default-metrics-tls" not found Nov 04 00:56:57.356906 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.356808 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 00:56:57.356906 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:56:57.356841 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert podName:b64026c1-aa80-4092-829e-91e5e8164afe nodeName:}" failed. No retries permitted until 2026-11-04 00:57:01.356831232 +0000 UTC m=+40.087079402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wkh6k" (UID: "b64026c1-aa80-4092-829e-91e5e8164afe") : secret "networking-console-plugin-cert" not found Nov 04 00:56:58.023798 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:58.022825 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc" event={"ID":"2fa177a8-889a-4a1e-9616-7d2f81a55c8b","Type":"ContainerStarted","Data":"9d366684be69a3184a92bb0f7d52598a4d823b887a485018a96d463ec4300733"} Nov 04 00:56:58.146418 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:58.146354 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bnf5w" podStartSLOduration=460.146328413 podStartE2EDuration="7m40.146328413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:55.83301433 +0000 UTC m=+34.563262524" watchObservedRunningTime="2026-11-04 00:56:58.146328413 +0000 UTC m=+36.876576605" Nov 04 00:56:59.033422 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:59.031907 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/0.log" Nov 04 00:56:59.033422 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:59.031951 2577 generic.go:356] "Generic (PLEG): container finished" podID="e2534417-f751-4241-b22d-3ea465dc6a19" containerID="d73741a091dbb0348382956e2e455b715c477076caa55c4473fa97bc469c553d" exitCode=255 Nov 04 00:56:59.033422 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:59.032829 2577 scope.go:122] "RemoveContainer" containerID="d73741a091dbb0348382956e2e455b715c477076caa55c4473fa97bc469c553d" Nov 04 00:56:59.033422 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:59.033132 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" event={"ID":"e2534417-f751-4241-b22d-3ea465dc6a19","Type":"ContainerDied","Data":"d73741a091dbb0348382956e2e455b715c477076caa55c4473fa97bc469c553d"} Nov 04 00:56:59.160485 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:56:59.158840 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc" podStartSLOduration=425.340246985 podStartE2EDuration="7m8.158826534s" totalImagesPullingTime="2.818579549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:58.14844734 +0000 UTC m=+36.878695535" watchObservedRunningTime="2026-11-04 00:56:59.158826534 +0000 UTC m=+37.889074742" Nov 04 00:57:00.042104 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:00.042070 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/1.log" Nov 04 00:57:00.042636 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:00.042612 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/0.log" Nov 04 00:57:00.042738 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:00.042659 2577 generic.go:356] "Generic (PLEG): container finished" podID="e2534417-f751-4241-b22d-3ea465dc6a19" containerID="cdd4c6b01fe2d4592d5b5062a9b1a00fd0a2188fb0e2c58f0fd28da37c1d2822" exitCode=255 Nov 04 00:57:00.042908 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:00.042763 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" event={"ID":"e2534417-f751-4241-b22d-3ea465dc6a19","Type":"ContainerDied","Data":"cdd4c6b01fe2d4592d5b5062a9b1a00fd0a2188fb0e2c58f0fd28da37c1d2822"} Nov 04 00:57:00.042908 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:00.042835 2577 scope.go:122] "RemoveContainer" containerID="d73741a091dbb0348382956e2e455b715c477076caa55c4473fa97bc469c553d" Nov 04 00:57:00.043181 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:00.043164 2577 scope.go:122] "RemoveContainer" containerID="cdd4c6b01fe2d4592d5b5062a9b1a00fd0a2188fb0e2c58f0fd28da37c1d2822" Nov 04 00:57:00.043448 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:00.043415 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-478s5_openshift-console-operator(e2534417-f751-4241-b22d-3ea465dc6a19)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" podUID="e2534417-f751-4241-b22d-3ea465dc6a19" Nov 04 00:57:00.045461 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:00.045440 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" event={"ID":"576edbd7-09bc-44d8-ba89-12af92909132","Type":"ContainerStarted","Data":"6a8d058c0163889b8109e4ebf30494274cedbdb723f0731e08935371275e9daa"} Nov 04 00:57:00.293848 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:00.293177 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" podStartSLOduration=424.090871802 podStartE2EDuration="7m9.293158646s" totalImagesPullingTime="5.202286844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:00.292617577 +0000 UTC m=+39.022865771" watchObservedRunningTime="2026-11-04 00:57:00.293158646 +0000 UTC m=+39.023407050" Nov 04 00:57:01.050407 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.050381 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/1.log" Nov 04 00:57:01.050830 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.050783 2577 scope.go:122] "RemoveContainer" containerID="cdd4c6b01fe2d4592d5b5062a9b1a00fd0a2188fb0e2c58f0fd28da37c1d2822" Nov 04 00:57:01.051022 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.051000 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-478s5_openshift-console-operator(e2534417-f751-4241-b22d-3ea465dc6a19)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" podUID="e2534417-f751-4241-b22d-3ea465dc6a19" Nov 04 00:57:01.052313 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.052246 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-s85v2" event={"ID":"68909b0c-2aad-4e3f-ab79-d13d21561db5","Type":"ContainerStarted","Data":"a1e9fb5fadf90fc7141b0da5209f22e7a8b37a179098e0390d01b75890336dda"} Nov 04 00:57:01.053922 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.053900 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dwkfx" event={"ID":"40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e","Type":"ContainerStarted","Data":"0a644b920ea6434839d9e6d9a9fdbe414f849c009cc718a53e2d157b21b86894"} Nov 04 00:57:01.055523 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.055499 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6" event={"ID":"2248ed8a-1d14-4a43-8e03-7bf83ed903a6","Type":"ContainerStarted","Data":"9395a1825c41976951f9d5edcdcad2a15b1e4e52b2ecb50e5a1dd87fc0866f67"} Nov 04 00:57:01.102003 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.101972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:57:01.102185 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.102140 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:01.102230 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.102210 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls podName:21952e92-3601-4969-adbd-6c6ff9b87074 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:09.10219484 +0000 UTC m=+47.832443024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bql4" (UID: "21952e92-3601-4969-adbd-6c6ff9b87074") : secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:01.203231 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.203191 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:01.203432 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.203415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:01.203624 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.203578 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 00:57:01.203744 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.203662 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs podName:28aa6fa1-33f8-450e-a1f1-d382e7cf6a47 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:09.203641182 +0000 UTC m=+47.933889372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs") pod "router-default-767db4f785-j87sp" (UID: "28aa6fa1-33f8-450e-a1f1-d382e7cf6a47") : secret "router-metrics-certs-default" not found Nov 04 00:57:01.203822 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.203747 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle podName:28aa6fa1-33f8-450e-a1f1-d382e7cf6a47 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:09.203730485 +0000 UTC m=+47.933978669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle") pod "router-default-767db4f785-j87sp" (UID: "28aa6fa1-33f8-450e-a1f1-d382e7cf6a47") : configmap references non-existent config key: service-ca.crt Nov 04 00:57:01.239542 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.239510 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq"] Nov 04 00:57:01.254610 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.254579 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq" Nov 04 00:57:01.295847 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.295812 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq"] Nov 04 00:57:01.304252 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.304189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:01.304381 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.304310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:01.304381 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.304315 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:01.304381 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.304336 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-559546cdcc-tvmhg: secret "image-registry-tls" not found Nov 04 00:57:01.304381 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.304366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:57:01.304540 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.304392 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls podName:ac49addd-a050-4d12-bdc0-20345e446a15 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:09.304374482 +0000 UTC m=+48.034622664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls") pod "image-registry-559546cdcc-tvmhg" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15") : secret "image-registry-tls" not found Nov 04 00:57:01.304540 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.304449 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:57:01.304540 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.304454 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:01.304540 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.304474 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5768bf545b-g67zq: secret "image-registry-tls" not found Nov 04 00:57:01.304540 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.304508 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls podName:f84f08f4-67c2-48b5-b939-ab9269a43bfc nodeName:}" failed. No retries permitted until 2026-11-04 00:57:09.304494068 +0000 UTC m=+48.034742253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n4pvn" (UID: "f84f08f4-67c2-48b5-b939-ab9269a43bfc") : secret "samples-operator-tls" not found Nov 04 00:57:01.304540 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.304524 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls podName:cbd1d155-f232-4679-a70a-ac3911aeb391 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:09.304515616 +0000 UTC m=+48.034763784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls") pod "image-registry-5768bf545b-g67zq" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391") : secret "image-registry-tls" not found Nov 04 00:57:01.304961 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.304930 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dwkfx" podStartSLOduration=2.565670054 podStartE2EDuration="8.304918748s" totalImagesPullingTime="5.739248694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:01.303874194 +0000 UTC m=+40.034122387" watchObservedRunningTime="2026-11-04 00:57:01.304918748 +0000 UTC m=+40.035166938" Nov 04 00:57:01.405320 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.405295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh8cn\" (UniqueName: \"kubernetes.io/projected/f8fe96da-a52f-4f6c-ae1d-229955f4d9ec-kube-api-access-jh8cn\") pod \"migrator-66df5d496b-2whjq\" (UID: \"f8fe96da-a52f-4f6c-ae1d-229955f4d9ec\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq" Nov 04 00:57:01.405444 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.405337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:01.405499 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.405461 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:01.405499 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.405475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:57:01.405596 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.405508 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls podName:3bd0fcf4-1163-444a-95cd-be27096da34d nodeName:}" failed. No retries permitted until 2026-11-04 00:57:09.40549656 +0000 UTC m=+48.135744733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls") pod "dns-default-zm7jt" (UID: "3bd0fcf4-1163-444a-95cd-be27096da34d") : secret "dns-default-metrics-tls" not found Nov 04 00:57:01.405596 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.405557 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 00:57:01.405696 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:01.405598 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert podName:b64026c1-aa80-4092-829e-91e5e8164afe nodeName:}" failed. No retries permitted until 2026-11-04 00:57:09.405587902 +0000 UTC m=+48.135836072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wkh6k" (UID: "b64026c1-aa80-4092-829e-91e5e8164afe") : secret "networking-console-plugin-cert" not found Nov 04 00:57:01.407764 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.407749 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 04 00:57:01.408015 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.408003 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:01.408080 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.408066 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-p4bs4\"" Nov 04 00:57:01.431528 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.431483 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6" podStartSLOduration=423.969079825 podStartE2EDuration="7m10.431470397s" totalImagesPullingTime="6.462390572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:01.430195754 +0000 UTC m=+40.160443945" watchObservedRunningTime="2026-11-04 00:57:01.431470397 +0000 UTC m=+40.161718587" Nov 04 00:57:01.506611 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.506574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8cn\" (UniqueName: \"kubernetes.io/projected/f8fe96da-a52f-4f6c-ae1d-229955f4d9ec-kube-api-access-jh8cn\") pod \"migrator-66df5d496b-2whjq\" (UID: \"f8fe96da-a52f-4f6c-ae1d-229955f4d9ec\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq" Nov 04 00:57:01.554768 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.554675 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-s85v2" podStartSLOduration=423.912638133 podStartE2EDuration="7m10.554658453s" totalImagesPullingTime="6.64202032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:01.553507137 +0000 UTC m=+40.283755328" watchObservedRunningTime="2026-11-04 00:57:01.554658453 +0000 UTC m=+40.284906648" Nov 04 00:57:01.573467 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.573438 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8cn\" (UniqueName: \"kubernetes.io/projected/f8fe96da-a52f-4f6c-ae1d-229955f4d9ec-kube-api-access-jh8cn\") pod \"migrator-66df5d496b-2whjq\" (UID: \"f8fe96da-a52f-4f6c-ae1d-229955f4d9ec\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq" Nov 04 00:57:01.575218 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.575193 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq" Nov 04 00:57:01.970478 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:01.970444 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq"] Nov 04 00:57:01.973315 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:57:01.973289 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fe96da_a52f_4f6c_ae1d_229955f4d9ec.slice/crio-fce1cf781706c1c7811cbb7d7e8cb8d1afddefbdd292e8462c5c13f5b13645c2 WatchSource:0}: Error finding container fce1cf781706c1c7811cbb7d7e8cb8d1afddefbdd292e8462c5c13f5b13645c2: Status 404 returned error can't find the container with id fce1cf781706c1c7811cbb7d7e8cb8d1afddefbdd292e8462c5c13f5b13645c2 Nov 04 00:57:02.060587 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.060545 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" event={"ID":"b2095b82-50e2-40c5-804e-4f91d84d5e03","Type":"ContainerStarted","Data":"c389b890606a785712a9bb6372663c8315f4248618d7e87669006235b10ecae1"} Nov 04 00:57:02.062160 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.062131 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9slf7" event={"ID":"7a4f60af-259e-442f-a17b-911beb60fee5","Type":"ContainerStarted","Data":"521196ff40ce7698bbc7954a21e386518717cfb90280de0d700a0ede4f9f3632"} Nov 04 00:57:02.062294 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.062188 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:57:02.063473 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.063446 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq" event={"ID":"f8fe96da-a52f-4f6c-ae1d-229955f4d9ec","Type":"ContainerStarted","Data":"fce1cf781706c1c7811cbb7d7e8cb8d1afddefbdd292e8462c5c13f5b13645c2"} Nov 04 00:57:02.187912 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.187860 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" podStartSLOduration=424.066655696 podStartE2EDuration="7m11.187845041s" totalImagesPullingTime="7.121189345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:02.18605564 +0000 UTC m=+40.916303838" watchObservedRunningTime="2026-11-04 00:57:02.187845041 +0000 UTC m=+40.918093232" Nov 04 00:57:02.252654 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.252586 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w6x6k"] Nov 04 00:57:02.275652 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.275623 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.309910 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.309880 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w6x6k"] Nov 04 00:57:02.312281 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.312235 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9slf7" podStartSLOduration=34.483653872 podStartE2EDuration="40.312223383s" totalImagesPullingTime="5.828569511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:02.311480786 +0000 UTC m=+41.041728977" watchObservedRunningTime="2026-11-04 00:57:02.312223383 +0000 UTC m=+41.042471574" Nov 04 00:57:02.415188 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.415153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f5b81fe1-bb35-4200-9b69-c49ed020ca64-crio-socket\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.415345 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.415199 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.415345 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.415285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5b81fe1-bb35-4200-9b69-c49ed020ca64-data-volume\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.415419 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.415396 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thr9\" (UniqueName: \"kubernetes.io/projected/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-api-access-7thr9\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.415500 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.415484 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.421621 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.421600 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 00:57:02.421956 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.421942 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7knl\"" Nov 04 00:57:02.421996 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.421941 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 00:57:02.515970 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.515907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.515970 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.515952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f5b81fe1-bb35-4200-9b69-c49ed020ca64-crio-socket\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.516161 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.516009 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f5b81fe1-bb35-4200-9b69-c49ed020ca64-crio-socket\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.516161 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:02.516048 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:02.516161 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:02.516109 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls podName:f5b81fe1-bb35-4200-9b69-c49ed020ca64 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:03.016090969 +0000 UTC m=+41.746339154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w6x6k" (UID: "f5b81fe1-bb35-4200-9b69-c49ed020ca64") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:02.516161 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.516158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.516313 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.516183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5b81fe1-bb35-4200-9b69-c49ed020ca64-data-volume\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.516347 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:02.516317 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 00:57:02.516397 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:02.516386 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm podName:f5b81fe1-bb35-4200-9b69-c49ed020ca64 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:03.016371559 +0000 UTC m=+41.746619732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w6x6k" (UID: "f5b81fe1-bb35-4200-9b69-c49ed020ca64") : configmap "kube-rbac-proxy" not found Nov 04 00:57:02.516435 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.516404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thr9\" (UniqueName: \"kubernetes.io/projected/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-api-access-7thr9\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.516474 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.516446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5b81fe1-bb35-4200-9b69-c49ed020ca64-data-volume\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:02.578425 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:02.578391 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thr9\" (UniqueName: \"kubernetes.io/projected/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-api-access-7thr9\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:03.021299 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:03.021257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:03.021504 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:03.021352 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:03.021504 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:03.021415 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:03.021504 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:03.021462 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 00:57:03.021630 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:03.021526 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls podName:f5b81fe1-bb35-4200-9b69-c49ed020ca64 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:04.021507707 +0000 UTC m=+42.751755879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w6x6k" (UID: "f5b81fe1-bb35-4200-9b69-c49ed020ca64") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:03.021630 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:03.021556 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm podName:f5b81fe1-bb35-4200-9b69-c49ed020ca64 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:04.021545489 +0000 UTC m=+42.751793661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w6x6k" (UID: "f5b81fe1-bb35-4200-9b69-c49ed020ca64") : configmap "kube-rbac-proxy" not found Nov 04 00:57:03.784396 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:03.784358 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:03.784902 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:03.784885 2577 scope.go:122] "RemoveContainer" containerID="cdd4c6b01fe2d4592d5b5062a9b1a00fd0a2188fb0e2c58f0fd28da37c1d2822" Nov 04 00:57:03.785169 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:03.785112 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-478s5_openshift-console-operator(e2534417-f751-4241-b22d-3ea465dc6a19)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" podUID="e2534417-f751-4241-b22d-3ea465dc6a19" Nov 04 00:57:03.788912 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:03.788891 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:04.031554 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:04.031519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:04.031743 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:04.031586 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:04.031743 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:04.031693 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 00:57:04.031743 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:04.031700 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:04.031743 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:04.031746 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm podName:f5b81fe1-bb35-4200-9b69-c49ed020ca64 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:06.031734287 +0000 UTC m=+44.761982455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w6x6k" (UID: "f5b81fe1-bb35-4200-9b69-c49ed020ca64") : configmap "kube-rbac-proxy" not found Nov 04 00:57:04.031939 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:04.031770 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls podName:f5b81fe1-bb35-4200-9b69-c49ed020ca64 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:06.031752054 +0000 UTC m=+44.762000224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w6x6k" (UID: "f5b81fe1-bb35-4200-9b69-c49ed020ca64") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:04.074224 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:04.072081 2577 scope.go:122] "RemoveContainer" containerID="cdd4c6b01fe2d4592d5b5062a9b1a00fd0a2188fb0e2c58f0fd28da37c1d2822" Nov 04 00:57:04.074224 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:04.072436 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-478s5_openshift-console-operator(e2534417-f751-4241-b22d-3ea465dc6a19)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" podUID="e2534417-f751-4241-b22d-3ea465dc6a19" Nov 04 00:57:05.076157 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:05.076107 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq" event={"ID":"f8fe96da-a52f-4f6c-ae1d-229955f4d9ec","Type":"ContainerStarted","Data":"c017eda03b485d1eb50e22812963c0c9d162ee650b284dc15f1499ddd24c4116"} Nov 04 00:57:06.052077 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:06.052045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:06.052284 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:06.052110 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:06.052284 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:06.052224 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:06.052396 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:06.052288 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls podName:f5b81fe1-bb35-4200-9b69-c49ed020ca64 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:10.05227342 +0000 UTC m=+48.782521588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w6x6k" (UID: "f5b81fe1-bb35-4200-9b69-c49ed020ca64") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:06.052733 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:06.052707 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:09.182936 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:09.182884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:57:09.183468 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.183046 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:09.183468 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.183154 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls podName:21952e92-3601-4969-adbd-6c6ff9b87074 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:25.183113724 +0000 UTC m=+63.913361897 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bql4" (UID: "21952e92-3601-4969-adbd-6c6ff9b87074") : secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:09.283993 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:09.283958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:09.284211 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:09.284074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:09.284211 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.284165 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle podName:28aa6fa1-33f8-450e-a1f1-d382e7cf6a47 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:25.284142959 +0000 UTC m=+64.014391130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle") pod "router-default-767db4f785-j87sp" (UID: "28aa6fa1-33f8-450e-a1f1-d382e7cf6a47") : configmap references non-existent config key: service-ca.crt Nov 04 00:57:09.284211 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.284210 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 00:57:09.284388 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.284252 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs podName:28aa6fa1-33f8-450e-a1f1-d382e7cf6a47 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:25.284241888 +0000 UTC m=+64.014490057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs") pod "router-default-767db4f785-j87sp" (UID: "28aa6fa1-33f8-450e-a1f1-d382e7cf6a47") : secret "router-metrics-certs-default" not found Nov 04 00:57:09.385046 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:09.385001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:09.385046 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:09.385046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:09.385302 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.385172 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:09.385302 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.385183 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:09.385302 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:09.385197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:57:09.385302 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.385205 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-559546cdcc-tvmhg: secret "image-registry-tls" not found Nov 04 00:57:09.385415 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.385343 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls podName:ac49addd-a050-4d12-bdc0-20345e446a15 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:25.385331087 +0000 UTC m=+64.115579260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls") pod "image-registry-559546cdcc-tvmhg" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15") : secret "image-registry-tls" not found Nov 04 00:57:09.385415 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.385188 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5768bf545b-g67zq: secret "image-registry-tls" not found Nov 04 00:57:09.385415 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.385374 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls podName:cbd1d155-f232-4679-a70a-ac3911aeb391 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:25.385367611 +0000 UTC m=+64.115615784 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls") pod "image-registry-5768bf545b-g67zq" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391") : secret "image-registry-tls" not found Nov 04 00:57:09.385693 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.385677 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:57:09.385742 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.385733 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls podName:f84f08f4-67c2-48b5-b939-ab9269a43bfc nodeName:}" failed. No retries permitted until 2026-11-04 00:57:25.385720471 +0000 UTC m=+64.115968641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n4pvn" (UID: "f84f08f4-67c2-48b5-b939-ab9269a43bfc") : secret "samples-operator-tls" not found Nov 04 00:57:09.486620 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:09.486537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:57:09.486765 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:09.486650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:09.486765 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.486682 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 00:57:09.486765 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.486745 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert podName:b64026c1-aa80-4092-829e-91e5e8164afe nodeName:}" failed. No retries permitted until 2026-11-04 00:57:25.486732909 +0000 UTC m=+64.216981078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wkh6k" (UID: "b64026c1-aa80-4092-829e-91e5e8164afe") : secret "networking-console-plugin-cert" not found Nov 04 00:57:09.486885 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.486773 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:09.486885 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:09.486810 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls podName:3bd0fcf4-1163-444a-95cd-be27096da34d nodeName:}" failed. No retries permitted until 2026-11-04 00:57:25.486800522 +0000 UTC m=+64.217048708 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls") pod "dns-default-zm7jt" (UID: "3bd0fcf4-1163-444a-95cd-be27096da34d") : secret "dns-default-metrics-tls" not found Nov 04 00:57:10.092843 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:10.092810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:10.093023 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:10.092958 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:10.093064 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:10.093025 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls podName:f5b81fe1-bb35-4200-9b69-c49ed020ca64 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:18.093009308 +0000 UTC m=+56.823257491 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w6x6k" (UID: "f5b81fe1-bb35-4200-9b69-c49ed020ca64") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:10.272886 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:10.272860 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rvxjh_011d3a1a-a2a5-4ef3-a871-453fb22817d5/dns-node-resolver/0.log" Nov 04 00:57:11.273661 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:11.273636 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vbxr9_762f4cf8-0912-4663-94e4-ef0c84740817/node-ca/0.log" Nov 04 00:57:11.343038 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:11.342998 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7smhm_d15630f9-042e-4c31-8f70-c8733c9dc31e/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:11.673499 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:11.673468 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dwkfx_40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e/serve-healthcheck-canary/0.log" Nov 04 00:57:11.783569 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:11.783532 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7smhm_d15630f9-042e-4c31-8f70-c8733c9dc31e/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:11.800075 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:11.800045 2577 scope.go:122] "RemoveContainer" containerID="cdd4c6b01fe2d4592d5b5062a9b1a00fd0a2188fb0e2c58f0fd28da37c1d2822" Nov 04 00:57:11.995729 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:11.995647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7smhm_d15630f9-042e-4c31-8f70-c8733c9dc31e/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:12.098965 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:12.098937 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 00:57:12.099401 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:12.099385 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/1.log" Nov 04 00:57:12.099461 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:12.099415 2577 generic.go:356] "Generic (PLEG): container finished" podID="e2534417-f751-4241-b22d-3ea465dc6a19" containerID="950227cc5d7e90c9d8958582b2659aeccb3e10e1eb241adbb3125cc62fba7272" exitCode=255 Nov 04 00:57:12.099498 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:12.099471 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" event={"ID":"e2534417-f751-4241-b22d-3ea465dc6a19","Type":"ContainerDied","Data":"950227cc5d7e90c9d8958582b2659aeccb3e10e1eb241adbb3125cc62fba7272"} Nov 04 00:57:12.099547 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:12.099519 2577 scope.go:122] "RemoveContainer" containerID="cdd4c6b01fe2d4592d5b5062a9b1a00fd0a2188fb0e2c58f0fd28da37c1d2822" Nov 04 00:57:12.099827 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:12.099808 2577 scope.go:122] "RemoveContainer" containerID="950227cc5d7e90c9d8958582b2659aeccb3e10e1eb241adbb3125cc62fba7272" Nov 04 00:57:12.100032 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:12.100011 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-478s5_openshift-console-operator(e2534417-f751-4241-b22d-3ea465dc6a19)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" podUID="e2534417-f751-4241-b22d-3ea465dc6a19" Nov 04 00:57:12.222934 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:12.222885 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq" podStartSLOduration=8.852755081 podStartE2EDuration="11.222870484s" totalImagesPullingTime="2.370115403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:05.20690788 +0000 UTC m=+43.937156071" watchObservedRunningTime="2026-11-04 00:57:12.222870484 +0000 UTC m=+50.953118672" Nov 04 00:57:12.273845 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:12.273767 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-2whjq_f8fe96da-a52f-4f6c-ae1d-229955f4d9ec/migrator/0.log" Nov 04 00:57:12.676200 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:12.676164 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh_576edbd7-09bc-44d8-ba89-12af92909132/kube-storage-version-migrator-operator/0.log" Nov 04 00:57:13.104521 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:13.104496 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 00:57:13.784854 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:13.784821 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:13.785265 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:13.785181 2577 scope.go:122] "RemoveContainer" containerID="950227cc5d7e90c9d8958582b2659aeccb3e10e1eb241adbb3125cc62fba7272" Nov 04 00:57:13.785373 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:13.785353 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-478s5_openshift-console-operator(e2534417-f751-4241-b22d-3ea465dc6a19)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" podUID="e2534417-f751-4241-b22d-3ea465dc6a19" Nov 04 00:57:13.789335 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:13.789320 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:14.108042 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:14.108019 2577 scope.go:122] "RemoveContainer" containerID="950227cc5d7e90c9d8958582b2659aeccb3e10e1eb241adbb3125cc62fba7272" Nov 04 00:57:14.108252 ip-10-0-136-94 kubenswrapper[2577]: E1104 00:57:14.108232 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-478s5_openshift-console-operator(e2534417-f751-4241-b22d-3ea465dc6a19)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" podUID="e2534417-f751-4241-b22d-3ea465dc6a19" Nov 04 00:57:18.161902 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:18.161866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:18.164237 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:18.164214 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:18.228985 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:18.228947 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:18.595638 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:18.595602 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w6x6k"] Nov 04 00:57:18.628831 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:57:18.628800 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b81fe1_bb35_4200_9b69_c49ed020ca64.slice/crio-4f5f2f4cd50771092c6b4f72faa2f4f864c51027da8fe854402d62837b1d4f1d WatchSource:0}: Error finding container 4f5f2f4cd50771092c6b4f72faa2f4f864c51027da8fe854402d62837b1d4f1d: Status 404 returned error can't find the container with id 4f5f2f4cd50771092c6b4f72faa2f4f864c51027da8fe854402d62837b1d4f1d Nov 04 00:57:19.124789 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:19.124759 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w6x6k" event={"ID":"f5b81fe1-bb35-4200-9b69-c49ed020ca64","Type":"ContainerStarted","Data":"88c4702fddc3f7c8719df8b35e0067eb0024ef48f7659587992a49db4c1b398e"} Nov 04 00:57:19.124956 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:19.124812 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w6x6k" event={"ID":"f5b81fe1-bb35-4200-9b69-c49ed020ca64","Type":"ContainerStarted","Data":"4f5f2f4cd50771092c6b4f72faa2f4f864c51027da8fe854402d62837b1d4f1d"} Nov 04 00:57:20.103978 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:20.103952 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:20.132159 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:20.132107 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w6x6k" event={"ID":"f5b81fe1-bb35-4200-9b69-c49ed020ca64","Type":"ContainerStarted","Data":"eb4a3ce30e6e89227d61a055902449d6910cc485bb8c52ee137a43315dfaced1"} Nov 04 00:57:23.147462 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:23.147427 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w6x6k" event={"ID":"f5b81fe1-bb35-4200-9b69-c49ed020ca64","Type":"ContainerStarted","Data":"2a9e532945749016283428ee509f8a8c4fc6de2c55ba219c601da68913a2886a"} Nov 04 00:57:25.225246 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.225194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:57:25.227595 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.227570 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:57:25.326083 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.326043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:25.326310 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.326237 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:25.326694 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.326670 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:25.328831 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.328805 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:25.427483 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.427452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:25.427483 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.427486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:25.427707 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.427522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:57:25.430034 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.430004 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:25.430172 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.430137 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:25.430172 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.430145 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:57:25.433866 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.433824 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w6x6k" podStartSLOduration=20.022771622 podStartE2EDuration="23.433810862s" totalImagesPullingTime="3.41103924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:23.271856783 +0000 UTC m=+62.002104973" watchObservedRunningTime="2026-11-04 00:57:25.433810862 +0000 UTC m=+64.164059052" Nov 04 00:57:25.434502 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.434486 2577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-559546cdcc-tvmhg"] Nov 04 00:57:25.491104 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.491026 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w95zk\"" Nov 04 00:57:25.495156 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.495134 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:25.528308 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.528277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:57:25.528480 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.528346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:25.530985 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.530823 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:25.531826 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.531799 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:57:25.547046 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.547023 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2cfvb\"" Nov 04 00:57:25.551874 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.551639 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:57:25.598686 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.598653 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9rklk\"" Nov 04 00:57:25.602761 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.602468 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:25.617325 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.617303 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:57:25.658484 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.657645 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:25.770417 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.770338 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-djbdf\"" Nov 04 00:57:25.774361 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.774342 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:57:25.843257 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.843234 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8chh7\"" Nov 04 00:57:25.847141 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.847106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:25.966677 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.966624 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4"] Nov 04 00:57:25.968043 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:25.968024 2577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-559546cdcc-tvmhg"] Nov 04 00:57:26.007068 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:57:26.007039 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21952e92_3601_4969_adbd_6c6ff9b87074.slice/crio-509fdb2ef10fc63c144f71b866b04e9f5ac3a494b8fe14e343da6cabdcd96233 WatchSource:0}: Error finding container 509fdb2ef10fc63c144f71b866b04e9f5ac3a494b8fe14e343da6cabdcd96233: Status 404 returned error can't find the container with id 509fdb2ef10fc63c144f71b866b04e9f5ac3a494b8fe14e343da6cabdcd96233 Nov 04 00:57:26.008358 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:57:26.008332 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac49addd_a050_4d12_bdc0_20345e446a15.slice/crio-674fcbdbd6d5767b57e4719770ebe1b2a124ae6ccc3463fefe335beaa79a91d4 WatchSource:0}: Error finding container 674fcbdbd6d5767b57e4719770ebe1b2a124ae6ccc3463fefe335beaa79a91d4: Status 404 returned error can't find the container with id 674fcbdbd6d5767b57e4719770ebe1b2a124ae6ccc3463fefe335beaa79a91d4 Nov 04 00:57:26.017922 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.017899 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn"] Nov 04 00:57:26.021169 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.021105 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-767db4f785-j87sp"] Nov 04 00:57:26.059815 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:57:26.059787 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28aa6fa1_33f8_450e_a1f1_d382e7cf6a47.slice/crio-4f2e5d44e6a63a5db33529802c38926684727b226ee5d611619d8cace6df161c WatchSource:0}: Error finding container 4f2e5d44e6a63a5db33529802c38926684727b226ee5d611619d8cace6df161c: Status 404 returned error can't find the container with id 4f2e5d44e6a63a5db33529802c38926684727b226ee5d611619d8cace6df161c Nov 04 00:57:26.075627 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.075601 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5768bf545b-g67zq"] Nov 04 00:57:26.078735 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:57:26.078714 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd1d155_f232_4679_a70a_ac3911aeb391.slice/crio-2189e299abfb14d8bc685d7a52518f27aaf275387ddbc8cacafd793743f4f5ef WatchSource:0}: Error finding container 2189e299abfb14d8bc685d7a52518f27aaf275387ddbc8cacafd793743f4f5ef: Status 404 returned error can't find the container with id 2189e299abfb14d8bc685d7a52518f27aaf275387ddbc8cacafd793743f4f5ef Nov 04 00:57:26.142192 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.142166 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wkh6k"] Nov 04 00:57:26.145304 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:57:26.145273 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64026c1_aa80_4092_829e_91e5e8164afe.slice/crio-1fe0e753a5e840cbc2828ee923374bdc8eb3f0e97ca6752d940d2fd231ce615d WatchSource:0}: Error finding container 1fe0e753a5e840cbc2828ee923374bdc8eb3f0e97ca6752d940d2fd231ce615d: Status 404 returned error can't find the container with id 1fe0e753a5e840cbc2828ee923374bdc8eb3f0e97ca6752d940d2fd231ce615d Nov 04 00:57:26.159373 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.159340 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" event={"ID":"b64026c1-aa80-4092-829e-91e5e8164afe","Type":"ContainerStarted","Data":"1fe0e753a5e840cbc2828ee923374bdc8eb3f0e97ca6752d940d2fd231ce615d"} Nov 04 00:57:26.160925 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.160899 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" event={"ID":"cbd1d155-f232-4679-a70a-ac3911aeb391","Type":"ContainerStarted","Data":"da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35"} Nov 04 00:57:26.161021 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.160946 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" event={"ID":"cbd1d155-f232-4679-a70a-ac3911aeb391","Type":"ContainerStarted","Data":"2189e299abfb14d8bc685d7a52518f27aaf275387ddbc8cacafd793743f4f5ef"} Nov 04 00:57:26.161021 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.160992 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:26.162439 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.162416 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" event={"ID":"f84f08f4-67c2-48b5-b939-ab9269a43bfc","Type":"ContainerStarted","Data":"45ffe87d163ac212b283931208beddac26db1866277950b6880a9c94be72cc14"} Nov 04 00:57:26.163660 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.163627 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" event={"ID":"21952e92-3601-4969-adbd-6c6ff9b87074","Type":"ContainerStarted","Data":"509fdb2ef10fc63c144f71b866b04e9f5ac3a494b8fe14e343da6cabdcd96233"} Nov 04 00:57:26.165176 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.165147 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" event={"ID":"ac49addd-a050-4d12-bdc0-20345e446a15","Type":"ContainerStarted","Data":"4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62"} Nov 04 00:57:26.165270 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.165198 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" event={"ID":"ac49addd-a050-4d12-bdc0-20345e446a15","Type":"ContainerStarted","Data":"674fcbdbd6d5767b57e4719770ebe1b2a124ae6ccc3463fefe335beaa79a91d4"} Nov 04 00:57:26.165270 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.165238 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:26.166726 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.166706 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-767db4f785-j87sp" event={"ID":"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47","Type":"ContainerStarted","Data":"181d801251786a95f969f3d500b0c81810738d6ceb238886f2203ddc5bae9bce"} Nov 04 00:57:26.166824 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.166744 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-767db4f785-j87sp" event={"ID":"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47","Type":"ContainerStarted","Data":"4f2e5d44e6a63a5db33529802c38926684727b226ee5d611619d8cace6df161c"} Nov 04 00:57:26.216247 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.216167 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zm7jt"] Nov 04 00:57:26.218593 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:57:26.218564 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd0fcf4_1163_444a_95cd_be27096da34d.slice/crio-6df800a919c45b98191e8c8a9d0256d8a7e8bacfe6c5ec83ec3f43ba56b3ea0d WatchSource:0}: Error finding container 6df800a919c45b98191e8c8a9d0256d8a7e8bacfe6c5ec83ec3f43ba56b3ea0d: Status 404 returned error can't find the container with id 6df800a919c45b98191e8c8a9d0256d8a7e8bacfe6c5ec83ec3f43ba56b3ea0d Nov 04 00:57:26.290606 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.290504 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" podStartSLOduration=56.290488644 podStartE2EDuration="56.290488644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:26.290434851 +0000 UTC m=+65.020683041" watchObservedRunningTime="2026-11-04 00:57:26.290488644 +0000 UTC m=+65.020736834" Nov 04 00:57:26.419690 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.419642 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-767db4f785-j87sp" podStartSLOduration=455.419629275 podStartE2EDuration="7m35.419629275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:26.419208055 +0000 UTC m=+65.149456246" watchObservedRunningTime="2026-11-04 00:57:26.419629275 +0000 UTC m=+65.149877460" Nov 04 00:57:26.550750 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.549994 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" podStartSLOduration=455.549973405 podStartE2EDuration="7m35.549973405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:26.546781184 +0000 UTC m=+65.277029386" watchObservedRunningTime="2026-11-04 00:57:26.549973405 +0000 UTC m=+65.280221596" Nov 04 00:57:26.603439 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.603400 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:26.606189 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.606155 2577 patch_prober.go:29] interesting pod/router-default-767db4f785-j87sp container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Nov 04 00:57:26.606189 ip-10-0-136-94 kubenswrapper[2577]: [+]has-synced ok Nov 04 00:57:26.606189 ip-10-0-136-94 kubenswrapper[2577]: [+]process-running ok Nov 04 00:57:26.606189 ip-10-0-136-94 kubenswrapper[2577]: healthz check failed Nov 04 00:57:26.606384 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.606218 2577 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-767db4f785-j87sp" podUID="28aa6fa1-33f8-450e-a1f1-d382e7cf6a47" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 04 00:57:26.739234 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.739197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:57:26.739398 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.739242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:57:26.795719 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.795686 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 00:57:26.795878 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.795783 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 00:57:26.802162 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.802090 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:57:26.802162 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:26.802102 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:57:27.011854 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:27.011822 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xq5nc\"" Nov 04 00:57:27.016019 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:27.015994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:57:27.060137 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:27.060060 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fh9wk\"" Nov 04 00:57:27.064692 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:27.064666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:57:27.171625 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:27.171583 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zm7jt" event={"ID":"3bd0fcf4-1163-444a-95cd-be27096da34d","Type":"ContainerStarted","Data":"6df800a919c45b98191e8c8a9d0256d8a7e8bacfe6c5ec83ec3f43ba56b3ea0d"} Nov 04 00:57:27.388169 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:27.388137 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pclxc"] Nov 04 00:57:27.392241 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:57:27.392202 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebe7f81_60bb_423b_a871_c2bab48db5c3.slice/crio-3e8768c7ee68de04c9ab28e87b723cbbce6e1c7ea8ba8d0bc205daa3f02c1c8c WatchSource:0}: Error finding container 3e8768c7ee68de04c9ab28e87b723cbbce6e1c7ea8ba8d0bc205daa3f02c1c8c: Status 404 returned error can't find the container with id 3e8768c7ee68de04c9ab28e87b723cbbce6e1c7ea8ba8d0bc205daa3f02c1c8c Nov 04 00:57:27.437680 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:27.437644 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z5rxm"] Nov 04 00:57:27.441619 ip-10-0-136-94 kubenswrapper[2577]: W1104 00:57:27.441567 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bd204f_98ed_42b0_9c0d_97baf9aa090d.slice/crio-c2a5a1afd44cdd3403fe3b2f1346ef9fe83fb402a511e722f18289c06bac754f WatchSource:0}: Error finding container c2a5a1afd44cdd3403fe3b2f1346ef9fe83fb402a511e722f18289c06bac754f: Status 404 returned error can't find the container with id c2a5a1afd44cdd3403fe3b2f1346ef9fe83fb402a511e722f18289c06bac754f Nov 04 00:57:27.605465 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:27.605434 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:28.178058 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:28.177974 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pclxc" event={"ID":"0ebe7f81-60bb-423b-a871-c2bab48db5c3","Type":"ContainerStarted","Data":"3e8768c7ee68de04c9ab28e87b723cbbce6e1c7ea8ba8d0bc205daa3f02c1c8c"} Nov 04 00:57:28.180246 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:28.180178 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z5rxm" event={"ID":"78bd204f-98ed-42b0-9c0d-97baf9aa090d","Type":"ContainerStarted","Data":"c2a5a1afd44cdd3403fe3b2f1346ef9fe83fb402a511e722f18289c06bac754f"} Nov 04 00:57:28.180850 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:28.180806 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:28.185358 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:28.185338 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:29.185684 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:29.185423 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" event={"ID":"b64026c1-aa80-4092-829e-91e5e8164afe","Type":"ContainerStarted","Data":"64518c8d94d20cfccb88ac49d70d9519d674880a1dbc3997deea8c1c4cadae5e"} Nov 04 00:57:29.188072 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:29.187997 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" event={"ID":"f84f08f4-67c2-48b5-b939-ab9269a43bfc","Type":"ContainerStarted","Data":"1971b1d5a0bca96e7da88a217e11cdad2421d458dddaa042be96e3812f8bcbe1"} Nov 04 00:57:29.188072 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:29.188061 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" event={"ID":"f84f08f4-67c2-48b5-b939-ab9269a43bfc","Type":"ContainerStarted","Data":"25a56c72ff8322c1791ad5d0345cb832578a019d9a8ee0df78b45baf18663c0b"} Nov 04 00:57:29.190566 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:29.190539 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" event={"ID":"21952e92-3601-4969-adbd-6c6ff9b87074","Type":"ContainerStarted","Data":"9909808a6a99710218b53b1c2b79eb144bfcad8ad41c5c11c46f1750d8f8537f"} Nov 04 00:57:29.310885 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:29.310837 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" podStartSLOduration=427.211539026 podStartE2EDuration="7m9.310823851s" totalImagesPullingTime="2.099284825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:29.308896217 +0000 UTC m=+68.039144409" watchObservedRunningTime="2026-11-04 00:57:29.310823851 +0000 UTC m=+68.041072042" Nov 04 00:57:29.438207 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:29.438104 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" podStartSLOduration=455.655284805 podStartE2EDuration="7m38.438085173s" totalImagesPullingTime="2.782800368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:29.437720734 +0000 UTC m=+68.167968938" watchObservedRunningTime="2026-11-04 00:57:29.438085173 +0000 UTC m=+68.168333366" Nov 04 00:57:29.570019 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:29.569947 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" podStartSLOduration=455.695743538 podStartE2EDuration="7m38.569927451s" totalImagesPullingTime="2.874183913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:29.567398771 +0000 UTC m=+68.297646963" watchObservedRunningTime="2026-11-04 00:57:29.569927451 +0000 UTC m=+68.300175643" Nov 04 00:57:30.197201 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:30.197168 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z5rxm" event={"ID":"78bd204f-98ed-42b0-9c0d-97baf9aa090d","Type":"ContainerStarted","Data":"fc03139f401daa12ca43950806e9751b560ac9b5cd116e1325ff0b3db583a5bc"} Nov 04 00:57:30.197612 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:30.197215 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z5rxm" event={"ID":"78bd204f-98ed-42b0-9c0d-97baf9aa090d","Type":"ContainerStarted","Data":"e90aec4a4fc62b27130d1721615b776c6204e8999dbe6c5c113ef775a3a36050"} Nov 04 00:57:30.321956 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:30.321912 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z5rxm" podStartSLOduration=67.51128976 podStartE2EDuration="1m9.321898381s" totalImagesPullingTime="1.810608621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:30.320583205 +0000 UTC m=+69.050831397" watchObservedRunningTime="2026-11-04 00:57:30.321898381 +0000 UTC m=+69.052146572" Nov 04 00:57:31.202790 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:31.202732 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zm7jt" event={"ID":"3bd0fcf4-1163-444a-95cd-be27096da34d","Type":"ContainerStarted","Data":"5500708d52cdd35356689cb7437fe241e012fb3a8cfc78ae4f9358130d88d022"} Nov 04 00:57:31.203235 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:31.202804 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zm7jt" event={"ID":"3bd0fcf4-1163-444a-95cd-be27096da34d","Type":"ContainerStarted","Data":"54dba20e3d02929c321fdb7e5844ba4ca96fd83579baf41ffd20d90a72faf69c"} Nov 04 00:57:31.332174 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:31.332102 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-zm7jt" podStartSLOduration=34.427717859 podStartE2EDuration="38.332086186s" totalImagesPullingTime="3.904368327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:31.3308835 +0000 UTC m=+70.061131691" watchObservedRunningTime="2026-11-04 00:57:31.332086186 +0000 UTC m=+70.062334415" Nov 04 00:57:32.206880 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:32.206839 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:33.071280 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:33.070659 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:57:34.216502 ip-10-0-136-94 kubenswrapper[2577]: I1104 00:57:34.216462 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pclxc" event={"ID":"0ebe7f81-60bb-423b-a871-c2bab48db5c3","Type":"ContainerStarted","Data":"66b8cf08992e7b386829809b500fda643bfa3734e9b99ead2ee4b1d80b7134bd"} Nov 04 00:57:34.291141 ip-10-0-136-94 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 00:57:34.304511 ip-10-0-136-94 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 00:57:34.304679 ip-10-0-136-94 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 00:57:34.304725 ip-10-0-136-94 systemd[1]: kubelet.service: Consumed 4.481s CPU time. Nov 04 00:57:34.315818 ip-10-0-136-94 systemd[1]: Starting Kubernetes Kubelet... Nov 04 00:57:34.414288 ip-10-0-136-94 kubenswrapper[5786]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:34.414288 ip-10-0-136-94 kubenswrapper[5786]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 00:57:34.414288 ip-10-0-136-94 kubenswrapper[5786]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:34.414288 ip-10-0-136-94 kubenswrapper[5786]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:34.416693 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416594 5786 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:34.416693 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416689 5786 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:34.416693 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416694 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:34.416693 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416699 5786 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416703 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416708 5786 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416712 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416718 5786 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416725 5786 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416729 5786 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416733 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416738 5786 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416743 5786 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416746 5786 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416750 5786 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416754 5786 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416760 5786 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416766 5786 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416770 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416775 5786 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416779 5786 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416790 5786 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:34.416968 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416794 5786 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416798 5786 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416802 5786 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416807 5786 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416811 5786 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416816 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416821 5786 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416826 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416831 5786 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416835 5786 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416840 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416844 5786 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416849 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416853 5786 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416857 5786 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416862 5786 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416866 5786 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416871 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416875 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:34.417824 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416880 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416884 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416888 5786 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416893 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416898 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416902 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416907 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416911 5786 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416916 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416920 5786 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416924 5786 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416928 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416933 5786 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416938 5786 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416942 5786 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416947 5786 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416952 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416957 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416962 5786 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:34.418700 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416966 5786 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416972 5786 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416976 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416981 5786 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416985 5786 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416990 5786 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416995 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.416999 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417006 5786 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417010 5786 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417015 5786 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417019 5786 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417025 5786 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417030 5786 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417035 5786 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417040 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417044 5786 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417049 5786 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417053 5786 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417057 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:34.419567 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417062 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417067 5786 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417071 5786 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417075 5786 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417079 5786 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417083 5786 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417088 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417092 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417097 5786 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417102 5786 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417107 5786 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417111 5786 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417132 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417137 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417141 5786 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417145 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417150 5786 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417154 5786 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417158 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417162 5786 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:34.420471 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417167 5786 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417172 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417176 5786 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417181 5786 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417185 5786 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.417191 5786 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418474 5786 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418485 5786 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418490 5786 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418495 5786 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418500 5786 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418504 5786 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418509 5786 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418514 5786 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418518 5786 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418522 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418527 5786 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418531 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418536 5786 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:34.421411 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418540 5786 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418544 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418549 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418554 5786 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418558 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418562 5786 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418566 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418570 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418575 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418579 5786 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418584 5786 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418588 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418593 5786 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418597 5786 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418601 5786 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418606 5786 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418610 5786 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418614 5786 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418618 5786 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418624 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:34.422305 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418629 5786 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418633 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418637 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418642 5786 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418646 5786 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418650 5786 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418654 5786 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418658 5786 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418662 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418667 5786 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418671 5786 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418675 5786 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418680 5786 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418684 5786 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418692 5786 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418698 5786 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418704 5786 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418709 5786 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418715 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:34.423231 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418720 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418725 5786 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418731 5786 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418736 5786 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418741 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418745 5786 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418750 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418754 5786 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418758 5786 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418762 5786 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418767 5786 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418772 5786 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418776 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418781 5786 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418786 5786 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418790 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418794 5786 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418799 5786 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418804 5786 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418808 5786 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:34.423782 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418813 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418817 5786 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418821 5786 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418825 5786 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418829 5786 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418834 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418838 5786 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418844 5786 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418848 5786 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418852 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418857 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418861 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418866 5786 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418870 5786 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418874 5786 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418879 5786 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418883 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418887 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418891 5786 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418896 5786 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:34.424304 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418900 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418904 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418908 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418912 5786 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418917 5786 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418923 5786 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418928 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418933 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418938 5786 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418942 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418946 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418950 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418954 5786 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.418959 5786 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419847 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419859 5786 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419865 5786 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419871 5786 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419876 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419880 5786 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:34.424840 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419885 5786 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419889 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419893 5786 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419898 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419902 5786 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419907 5786 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419912 5786 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419916 5786 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419920 5786 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419925 5786 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419929 5786 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419933 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419938 5786 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419942 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419947 5786 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419951 5786 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419955 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419962 5786 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419970 5786 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:34.425448 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419976 5786 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419980 5786 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419985 5786 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419989 5786 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419994 5786 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.419998 5786 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420002 5786 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420010 5786 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420014 5786 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420018 5786 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420023 5786 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420027 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420032 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420036 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420041 5786 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420045 5786 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420049 5786 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420054 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420058 5786 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:34.425918 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420063 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420067 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420072 5786 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420077 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420081 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420086 5786 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420090 5786 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420094 5786 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420098 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420103 5786 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420108 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420112 5786 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420133 5786 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420139 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420144 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420148 5786 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420153 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420157 5786 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420162 5786 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:34.426453 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420166 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420172 5786 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420176 5786 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420181 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420185 5786 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420189 5786 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420194 5786 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420198 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420202 5786 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420207 5786 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420211 5786 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420215 5786 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420219 5786 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420224 5786 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420228 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420232 5786 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420239 5786 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420243 5786 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420247 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420253 5786 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:34.426921 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420257 5786 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420262 5786 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420266 5786 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420270 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420274 5786 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420278 5786 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420283 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420295 5786 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420302 5786 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420306 5786 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420312 5786 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420316 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420321 5786 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420327 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420332 5786 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420336 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420340 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420344 5786 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420349 5786 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420353 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:34.427438 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420357 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420363 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.420368 5786 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420498 5786 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420510 5786 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420521 5786 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420528 5786 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420535 5786 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420543 5786 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420550 5786 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420557 5786 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420563 5786 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420568 5786 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420574 5786 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420579 5786 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420584 5786 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420589 5786 flags.go:64] FLAG: --cgroup-root="" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420594 5786 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420599 5786 flags.go:64] FLAG: --client-ca-file="" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420604 5786 flags.go:64] FLAG: --cloud-provider="external" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420609 5786 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420616 5786 flags.go:64] FLAG: --cluster-domain="" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420621 5786 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420626 5786 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 00:57:34.427936 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420631 5786 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420638 5786 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420645 5786 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420650 5786 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420655 5786 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420661 5786 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420665 5786 flags.go:64] FLAG: --contention-profiling="false" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420670 5786 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420675 5786 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420681 5786 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420686 5786 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420693 5786 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420698 5786 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420704 5786 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420708 5786 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420713 5786 flags.go:64] FLAG: --enable-server="true" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420720 5786 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420728 5786 flags.go:64] FLAG: --event-burst="100" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420733 5786 flags.go:64] FLAG: --event-qps="50" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420738 5786 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420743 5786 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420748 5786 flags.go:64] FLAG: --eviction-hard="" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420754 5786 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420759 5786 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420764 5786 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 00:57:34.428676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420769 5786 flags.go:64] FLAG: --eviction-soft="" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420775 5786 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420780 5786 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420785 5786 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420791 5786 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420795 5786 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420800 5786 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420805 5786 flags.go:64] FLAG: --feature-gates="" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420813 5786 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420819 5786 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420825 5786 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420830 5786 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420835 5786 flags.go:64] FLAG: --healthz-port="10248" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420840 5786 flags.go:64] FLAG: --help="false" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420845 5786 flags.go:64] FLAG: --hostname-override="ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420850 5786 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420855 5786 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420860 5786 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420866 5786 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420872 5786 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420877 5786 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420881 5786 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420886 5786 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 00:57:34.429681 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420892 5786 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420897 5786 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420902 5786 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420907 5786 flags.go:64] FLAG: --kube-reserved="" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420913 5786 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420917 5786 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420923 5786 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420927 5786 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420932 5786 flags.go:64] FLAG: --lock-file="" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420937 5786 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420941 5786 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420946 5786 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420956 5786 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420961 5786 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420966 5786 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420971 5786 flags.go:64] FLAG: --logging-format="text" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420976 5786 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420984 5786 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420988 5786 flags.go:64] FLAG: --manifest-url="" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.420993 5786 flags.go:64] FLAG: --manifest-url-header="" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421001 5786 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421007 5786 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421013 5786 flags.go:64] FLAG: --max-pods="110" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421018 5786 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421023 5786 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 00:57:34.430309 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421027 5786 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421032 5786 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421037 5786 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421043 5786 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421048 5786 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421059 5786 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421064 5786 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421071 5786 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421076 5786 flags.go:64] FLAG: --pod-cidr="" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421081 5786 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421086 5786 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421091 5786 flags.go:64] FLAG: --pods-per-core="0" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421096 5786 flags.go:64] FLAG: --port="10250" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421101 5786 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421105 5786 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-07eb3e70498f5ac31" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421111 5786 flags.go:64] FLAG: --qos-reserved="" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421132 5786 flags.go:64] FLAG: --read-only-port="10255" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421137 5786 flags.go:64] FLAG: --register-node="true" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421142 5786 flags.go:64] FLAG: --register-with-taints="" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421149 5786 flags.go:64] FLAG: --registry-burst="10" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421154 5786 flags.go:64] FLAG: --registry-qps="5" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421159 5786 flags.go:64] FLAG: --reserved-cpus="" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421164 5786 flags.go:64] FLAG: --reserved-memory="" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421171 5786 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421178 5786 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421183 5786 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 00:57:34.430950 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421188 5786 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421193 5786 flags.go:64] FLAG: --runonce="false" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421198 5786 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421203 5786 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421209 5786 flags.go:64] FLAG: --seccomp-default="false" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421213 5786 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421218 5786 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421223 5786 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421229 5786 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421233 5786 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421238 5786 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421243 5786 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421248 5786 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421255 5786 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421260 5786 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421265 5786 flags.go:64] FLAG: --system-cgroups="" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421270 5786 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421279 5786 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421284 5786 flags.go:64] FLAG: --tls-cert-file="" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421289 5786 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421296 5786 flags.go:64] FLAG: --tls-min-version="" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421300 5786 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421305 5786 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421310 5786 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421314 5786 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 00:57:34.431612 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421319 5786 flags.go:64] FLAG: --v="2" Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421326 5786 flags.go:64] FLAG: --version="false" Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421333 5786 flags.go:64] FLAG: --vmodule="" Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421339 5786 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.421354 5786 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421608 5786 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421616 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421622 5786 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421626 5786 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421631 5786 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421635 5786 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421640 5786 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421644 5786 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421648 5786 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421653 5786 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421658 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421662 5786 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421667 5786 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421671 5786 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421676 5786 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421682 5786 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:34.432269 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421687 5786 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421691 5786 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421695 5786 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421700 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421704 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421709 5786 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421714 5786 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421718 5786 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421722 5786 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421727 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421731 5786 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421735 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421739 5786 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421743 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421748 5786 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421752 5786 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421766 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421771 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421775 5786 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421780 5786 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:34.432828 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421784 5786 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421788 5786 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421793 5786 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421798 5786 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421802 5786 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421807 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421811 5786 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421815 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421820 5786 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421824 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421828 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421834 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421839 5786 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421843 5786 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421847 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421852 5786 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421856 5786 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421860 5786 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421864 5786 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:34.433414 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421869 5786 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421873 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421877 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421881 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421886 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421890 5786 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421895 5786 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421900 5786 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421904 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421912 5786 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421924 5786 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421931 5786 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421938 5786 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421943 5786 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421949 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421953 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421958 5786 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421963 5786 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421967 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:34.434070 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421971 5786 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421976 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421980 5786 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421985 5786 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421989 5786 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.421998 5786 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422005 5786 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422010 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422014 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422019 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422023 5786 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422027 5786 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422032 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422036 5786 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422041 5786 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422045 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422049 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422054 5786 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422058 5786 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:34.434668 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422062 5786 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422067 5786 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422071 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422077 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422082 5786 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422087 5786 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422092 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422096 5786 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422100 5786 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422104 5786 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422109 5786 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422113 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422134 5786 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.422177 5786 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422253 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422261 5786 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:34.435201 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422266 5786 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422271 5786 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422279 5786 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422283 5786 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422288 5786 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422292 5786 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422296 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422300 5786 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422305 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422309 5786 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422313 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422317 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422322 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422326 5786 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422330 5786 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422340 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422345 5786 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422349 5786 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422353 5786 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:34.435602 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422358 5786 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422363 5786 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422367 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422379 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422384 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422388 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422392 5786 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422397 5786 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422402 5786 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422406 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422410 5786 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422422 5786 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422428 5786 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422433 5786 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422439 5786 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422447 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422452 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422457 5786 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422461 5786 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:34.436203 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422467 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422471 5786 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422475 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422479 5786 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422484 5786 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422488 5786 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422495 5786 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422501 5786 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422505 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422510 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422514 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422518 5786 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422522 5786 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422527 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422531 5786 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422536 5786 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422540 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422547 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422551 5786 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:34.436901 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422556 5786 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422561 5786 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422565 5786 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422570 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422574 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422580 5786 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422584 5786 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422588 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422593 5786 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422599 5786 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422603 5786 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422607 5786 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422612 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422616 5786 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422621 5786 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422626 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422630 5786 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422634 5786 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422639 5786 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:34.437428 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422644 5786 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422648 5786 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422652 5786 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422657 5786 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422661 5786 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422666 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422671 5786 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422675 5786 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422679 5786 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422684 5786 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422688 5786 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422692 5786 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422697 5786 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422702 5786 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422708 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422712 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422718 5786 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422722 5786 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422727 5786 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422731 5786 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:34.437903 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422735 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422739 5786 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422746 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422751 5786 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422755 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422760 5786 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422764 5786 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.422768 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.427082 5786 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.427098 5786 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.427135 5786 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.427143 5786 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427213 5786 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427219 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427222 5786 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427225 5786 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427228 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427231 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427234 5786 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:34.438563 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427237 5786 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427239 5786 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427242 5786 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427245 5786 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427248 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427256 5786 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427259 5786 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427262 5786 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427265 5786 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427268 5786 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427271 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427273 5786 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427276 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427279 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427281 5786 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427284 5786 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427287 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427289 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427292 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:34.439228 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427295 5786 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427297 5786 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427300 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427303 5786 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427306 5786 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427308 5786 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427311 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427314 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427317 5786 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427319 5786 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427322 5786 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427325 5786 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427329 5786 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427332 5786 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427335 5786 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427337 5786 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427340 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427343 5786 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427345 5786 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:34.439697 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427353 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427356 5786 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427360 5786 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427362 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427365 5786 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427367 5786 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427370 5786 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427373 5786 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427376 5786 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427379 5786 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427381 5786 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427384 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427387 5786 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427389 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427392 5786 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427394 5786 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427397 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427400 5786 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427402 5786 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427405 5786 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:34.440393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427407 5786 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427410 5786 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427412 5786 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427415 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427417 5786 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427420 5786 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427422 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427425 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427427 5786 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427430 5786 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427432 5786 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427435 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427437 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427445 5786 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427449 5786 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427452 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427454 5786 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427457 5786 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427459 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427462 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:34.441060 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427465 5786 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427467 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427470 5786 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427472 5786 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427475 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427478 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427480 5786 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427483 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427485 5786 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427488 5786 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427490 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427493 5786 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427495 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427504 5786 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427508 5786 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427512 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427516 5786 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427519 5786 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427521 5786 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:34.441762 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427524 5786 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427527 5786 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.427554 5786 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427761 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427767 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427770 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427773 5786 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427781 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427784 5786 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427787 5786 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427790 5786 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427792 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427795 5786 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427797 5786 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427800 5786 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427802 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:34.442393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427805 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427808 5786 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427810 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427813 5786 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427815 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427818 5786 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427821 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427823 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427826 5786 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427828 5786 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427831 5786 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427834 5786 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427836 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427839 5786 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427841 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427844 5786 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427847 5786 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427849 5786 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427851 5786 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:34.442846 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427855 5786 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427859 5786 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427862 5786 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427865 5786 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427868 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427876 5786 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427879 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427881 5786 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427884 5786 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427887 5786 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427890 5786 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427893 5786 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427895 5786 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427898 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427900 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427903 5786 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427906 5786 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427908 5786 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427911 5786 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427913 5786 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:34.443363 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427916 5786 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427918 5786 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427921 5786 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427923 5786 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427926 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427929 5786 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427931 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427934 5786 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427938 5786 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427942 5786 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427945 5786 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427948 5786 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427950 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427953 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427956 5786 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427958 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427961 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427964 5786 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427976 5786 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:34.443858 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427980 5786 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427982 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427985 5786 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427987 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427990 5786 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427993 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427995 5786 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.427998 5786 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428000 5786 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428003 5786 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428006 5786 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428008 5786 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428011 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428013 5786 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428016 5786 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428019 5786 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428021 5786 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428024 5786 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428026 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:34.444466 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428029 5786 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428031 5786 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428034 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428036 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428039 5786 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428042 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428044 5786 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428047 5786 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428049 5786 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428052 5786 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428055 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428057 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428061 5786 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428071 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428074 5786 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428076 5786 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:34.444972 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.428098 5786 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428167 5786 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428171 5786 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428174 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428177 5786 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428179 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428182 5786 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428185 5786 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428188 5786 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428191 5786 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428195 5786 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428198 5786 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428202 5786 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428205 5786 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428207 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428210 5786 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428213 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428216 5786 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428219 5786 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428221 5786 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:34.445384 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428224 5786 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428227 5786 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428229 5786 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428232 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428235 5786 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428237 5786 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428240 5786 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428243 5786 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428245 5786 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428250 5786 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428258 5786 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428261 5786 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428264 5786 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428266 5786 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428269 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428272 5786 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428277 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428280 5786 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428284 5786 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:34.445878 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428288 5786 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428291 5786 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428294 5786 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428297 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428299 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428302 5786 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428305 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428307 5786 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428310 5786 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428312 5786 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428315 5786 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428318 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428320 5786 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428323 5786 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428326 5786 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428328 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428331 5786 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428333 5786 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428336 5786 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428338 5786 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:34.446393 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428341 5786 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428344 5786 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428347 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428350 5786 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428396 5786 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428679 5786 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428690 5786 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428700 5786 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428708 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428720 5786 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428728 5786 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428736 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428743 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428750 5786 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428758 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428766 5786 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428773 5786 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428781 5786 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428796 5786 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:34.446942 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428804 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428812 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428820 5786 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428828 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428835 5786 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428842 5786 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428851 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428858 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428866 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428873 5786 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428880 5786 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428887 5786 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428901 5786 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428908 5786 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428916 5786 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428928 5786 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428935 5786 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.428943 5786 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.429002 5786 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.429009 5786 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:34.447544 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.429017 5786 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.429024 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.429037 5786 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.429045 5786 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.429060 5786 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.429068 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.429076 5786 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.429083 5786 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:34.429090 5786 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.429224 5786 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.430745 5786 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.430811 5786 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.431014 5786 server.go:1014] "Starting client certificate rotation" Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.431145 5786 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.431196 5786 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:51:21 +0000 UTC" deadline="2028-04-09 11:27:32.89317342 +0000 UTC" Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.431226 5786 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12538h29m58.461950564s" Nov 04 00:57:34.448429 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.431491 5786 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:57:34.449035 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.432593 5786 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:57:34.449035 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.433809 5786 log.go:25] "Validated CRI v1 runtime API" Nov 04 00:57:34.449035 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.435858 5786 log.go:25] "Validated CRI v1 image API" Nov 04 00:57:34.449035 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.436574 5786 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 00:57:34.449035 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.438918 5786 fs.go:135] Filesystem UUIDs: map[11e46120-6bcf-4b2d-b33b-787e37952bef:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 97c4d611-aa15-459d-9dfa-f8aca5eab90f:/dev/nvme0n1p3] Nov 04 00:57:34.449366 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.438944 5786 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0aa691c57a3767004d22bad031b6714c9202828dbc8ac4b0c4a9136a8bf024b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0aa691c57a3767004d22bad031b6714c9202828dbc8ac4b0c4a9136a8bf024b2/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fe0e753a5e840cbc2828ee923374bdc8eb3f0e97ca6752d940d2fd231ce615d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fe0e753a5e840cbc2828ee923374bdc8eb3f0e97ca6752d940d2fd231ce615d/userdata/shm major:0 minor:383 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2189e299abfb14d8bc685d7a52518f27aaf275387ddbc8cacafd793743f4f5ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2189e299abfb14d8bc685d7a52518f27aaf275387ddbc8cacafd793743f4f5ef/userdata/shm major:0 minor:381 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a2d786d3267c9a6dfe3c4e55d983cc405632b2f97a5ebcbb6188ca0af23333a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a2d786d3267c9a6dfe3c4e55d983cc405632b2f97a5ebcbb6188ca0af23333a/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fd08dc37586bbaccc3bc48aeaf69b34346ad767e38bae7882ea89edbe1f7563/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fd08dc37586bbaccc3bc48aeaf69b34346ad767e38bae7882ea89edbe1f7563/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3183dfa600ae4dc32d285f2fa478fba8384bc67ec4e8397aace58f7e1f90a6d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3183dfa600ae4dc32d285f2fa478fba8384bc67ec4e8397aace58f7e1f90a6d3/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3775001e4a4db27460ffabfcefcfb3ba324e51b0a0a952ae48ea5a258c0e45da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3775001e4a4db27460ffabfcefcfb3ba324e51b0a0a952ae48ea5a258c0e45da/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c454bd79a3e2867138ba42d7a6bd571d65d85986eca640be0d9902327f1ee9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c454bd79a3e2867138ba42d7a6bd571d65d85986eca640be0d9902327f1ee9e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e8768c7ee68de04c9ab28e87b723cbbce6e1c7ea8ba8d0bc205daa3f02c1c8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e8768c7ee68de04c9ab28e87b723cbbce6e1c7ea8ba8d0bc205daa3f02c1c8c/userdata/shm major:0 minor:425 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45ffe87d163ac212b283931208beddac26db1866277950b6880a9c94be72cc14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45ffe87d163ac212b283931208beddac26db1866277950b6880a9c94be72cc14/userdata/shm major:0 minor:379 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49b06b427edd7f210d0b64acff80f9410d003f962b4e1dc21585105e8b31c7e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49b06b427edd7f210d0b64acff80f9410d003f962b4e1dc21585105e8b31c7e3/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f2e5d44e6a63a5db33529802c38926684727b226ee5d611619d8cace6df161c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f2e5d44e6a63a5db33529802c38926684727b226ee5d611619d8cace6df161c/userdata/shm major:0 minor:377 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f37ebc5fcf7f274bdf12d6e3b5ea789795044ce760ffa663c6f5f5a54e681ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f37ebc5fcf7f274bdf12d6e3b5ea789795044ce760ffa663c6f5f5a54e681ff/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f5f2f4cd50771092c6b4f72faa2f4f864c51027da8fe854402d62837b1d4f1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f5f2f4cd50771092c6b4f72faa2f4f864c51027da8fe854402d62837b1d4f1d/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/509fdb2ef10fc63c144f71b866b04e9f5ac3a494b8fe14e343da6cabdcd96233/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/509fdb2ef10fc63c144f71b866b04e9f5ac3a494b8fe14e343da6cabdcd96233/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6239dd443759994df4c3cd47ab57a764dba24aed318f6e9c06b20f08f7a90ebb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6239dd443759994df4c3cd47ab57a764dba24aed318f6e9c06b20f08f7a90ebb/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/674fcbdbd6d5767b57e4719770ebe1b2a124ae6ccc3463fefe335beaa79a91d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/674fcbdbd6d5767b57e4719770ebe1b2a124ae6ccc3463fefe335beaa79a91d4/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6df800a919c45b98191e8c8a9d0256d8a7e8bacfe6c5ec83ec3f43ba56b3ea0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6df800a919c45b98191e8c8a9d0256d8a7e8bacfe6c5ec83ec3f43ba56b3ea0d/userdata/shm major:0 minor:385 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ea2674aeefe26451bfec13e94671d6b93ea1ddb36e50af31affbb530e658065/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ea2674aeefe26451bfec13e94671d6b93ea1ddb36e50af31affbb530e658065/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77ec6df6a03a3682953ac1001b9b484abf1f9f75d62f158da113f8245237946f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77ec6df6a03a3682953ac1001b9b484abf1f9f75d62f158da113f8245237946f/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f72adec94ce451d7d63fab7300af4f04d09c96e9f9eab63f87f971a3b599a95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f72adec94ce451d7d63fab7300af4f04d09c96e9f9eab63f87f971a3b599a95/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92fe1d699adb2132995a76d9f46052f682f1321c3b82552bcfae33f4bebc2886/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92fe1d699adb2132995a76d9f46052f682f1321c3b82552bcfae33f4bebc2886/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94b25173f5fcdf9544a823009ff9682def60a2f6ad23e1e41e2bdb9644325959/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94b25173f5fcdf9544a823009ff9682def60a2f6ad23e1e41e2bdb9644325959/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ffc847eeba4c623cb9e5255c1d81b432fad4ff7638531bdd0db0112025965b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ffc847eeba4c623cb9e5255c1d81b432fad4ff7638531bdd0db0112025965b5/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa76b32e7ca58dbeb41e98e9dc715191e66fa15a184fc75e80230e779914d124/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa76b32e7ca58dbeb41e98e9dc715191e66fa15a184fc75e80230e779914d124/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abdcb4619f16a6dd66cf8370e60cfcbd1618f909b34cad8d6dd00a0ea97b6691/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abdcb4619f16a6dd66cf8370e60cfcbd1618f909b34cad8d6dd00a0ea97b6691/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac1b456d6d1432d94931f8726a3c15cdbde7f3f8e2c8199bb875232786d24fe6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac1b456d6d1432d94931f8726a3c15cdbde7f3f8e2c8199bb875232786d24fe6/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae166c7a149d538c0a9bbd6f65b27c277ccf8d39ef6e3b16efed217fbc5fb74a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae166c7a149d538c0a9bbd6f65b27c277ccf8d39ef6e3b16efed217fbc5fb74a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2a5a1afd44cdd3403fe3b2f1346ef9fe83fb402a511e722f18289c06bac754f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2a5a1afd44cdd3403fe3b2f1346ef9fe83fb402a511e722f18289c06bac754f/userdata/shm major:0 minor:427 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6cd2148f8da7e3f2640a201e9a468c6b9592a186ac5ec4a477b4332b204f25f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6cd2148f8da7e3f2640a201e9a468c6b9592a186ac5ec4a477b4332b204f25f/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc001fd8e0377241c1f20264c489dbba015c928138f744f48deb41a1accad95c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc001fd8e0377241c1f20264c489dbba015c928138f744f48deb41a1accad95c/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fce1cf781706c1c7811cbb7d7e8cb8d1afddefbdd292e8462c5c13f5b13645c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fce1cf781706c1c7811cbb7d7e8cb8d1afddefbdd292e8462c5c13f5b13645c2/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/011d3a1a-a2a5-4ef3-a871-453fb22817d5/volumes/kubernetes.io~projected/kube-api-access-jljcp:{mountpoint:/var/lib/kubelet/pods/011d3a1a-a2a5-4ef3-a871-453fb22817d5/volumes/kubernetes.io~projected/kube-api-access-jljcp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ebe7f81-60bb-423b-a871-c2bab48db5c3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0ebe7f81-60bb-423b-a871-c2bab48db5c3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21952e92-3601-4969-adbd-6c6ff9b87074/volumes/kubernetes.io~projected/kube-api-access-2b9md:{mountpoint:/var/lib/kubelet/pods/21952e92-3601-4969-adbd-6c6ff9b87074/volumes/kubernetes.io~projected/kube-api-access-2b9md major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21952e92-3601-4969-adbd-6c6ff9b87074/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/21952e92-3601-4969-adbd-6c6ff9b87074/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2248ed8a-1d14-4a43-8e03-7bf83ed903a6/volumes/kubernetes.io~projected/kube-api-access-gq6kt:{mountpoint:/var/lib/kubelet/pods/2248ed8a-1d14-4a43-8e03-7bf83ed903a6/volumes/kubernetes.io~projected/kube-api-access-gq6kt major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/volumes/kubernetes.io~projected/kube-api-access-llcg6:{mountpoint:/var/lib/kubelet/pods/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/volumes/kubernetes.io~projected/kube-api-access-llcg6 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/volumes/kubernetes.io~secret/metrics-certs major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/volumes/kubernetes.io~secret/stats-auth major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ad9dfd7-097e-4491-a3b3-ba3c89a52992/volumes/kubernetes.io~projected/kube-api-access-h4p4d:{mountpoint:/var/lib/kubelet/pods/2ad9dfd7-097e-4491-a3b3-ba3c89a52992/volumes/kubernetes.io~projected/kube-api-access-h4p4d major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c89163b-5d76-4794-93bb-56840ed333d7/volumes/kubernetes.io~projected/kube-api-access-jc65g:{mountpoint:/var/lib/kubelet/pods/2c89163b-5d76-4794-93bb-56840ed333d7/volumes/kubernetes.io~projected/kube-api-access-jc65g major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fa177a8-889a-4a1e-9616-7d2f81a55c8b/volumes/kubernetes.io~projected/kube-api-access-g8s82:{mountpoint:/var/lib/kubelet/pods/2fa177a8-889a-4a1e-9616-7d2f81a55c8b/volumes/kubernetes.io~projected/kube-api-access-g8s82 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3181e3c4-6f85-4302-965b-672bbf7534ba/volumes/kubernetes.io~projected/kube-api-access-6zkfp:{mountpoint:/var/lib/kubelet/pods/3181e3c4-6f85-4302-965b-672bbf7534ba/volumes/kubernetes.io~projected/kube-api-access-6zkfp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bd0fcf4-1163-444a-95cd-be27096da34d/volumes/kubernetes.io~projected/kube-api-access-q44f4:{mountpoint:/var/lib/kubelet/pods/3bd0fcf4-1163-444a-95cd-be27096da34d/volumes/kubernetes.io~projected/kube-api-access-q44f4 major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bd0fcf4-1163-444a-95cd-be27096da34d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3bd0fcf4-1163-444a-95cd-be27096da34d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e/volumes/kubernetes.io~projected/kube-api-access-5wtnl:{mountpoint:/var/lib/kubelet/pods/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e/volumes/kubernetes.io~projected/kube-api-access-5wtnl major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e/volumes/kubernetes.io~secret/cert major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/576edbd7-09bc-44d8-ba89-12af92909132/volumes/kubernetes.io~projected/kube-api-access-74c2k:{mountpoint:/var/lib/kubelet/pods/576edbd7-09bc-44d8-ba89-12af92909132/volumes/kubernetes.io~projected/kube-api-access-74c2k major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/576edbd7-09bc-44d8-ba89-12af92909132/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/576edbd7-09bc-44d8-ba89-12af92909132/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/593868da-9895-4d98-9da1-4fa815c11674/volumes/kubernetes.io~projected/kube-api-access-psm57:{mountpoint:/var/lib/kubelet/pods/593868da-9895-4d98-9da1-4fa815c11674/volumes/kubernetes.io~projected/kube-api-access-psm57 major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68909b0c-2aad-4e3f-ab79-d13d21561db5/volumes/kubernetes.io~projected/kube-api-access-bvl2j:{mountpoint:/var/lib/kubelet/pods/68909b0c-2aad-4e3f-ab79-d13d21561db5/volumes/kubernetes.io~projected/kube-api-access-bvl2j major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68909b0c-2aad-4e3f-ab79-d13d21561db5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/68909b0c-2aad-4e3f-ab79-d13d21561db5/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a/volumes/kubernetes.io~projected/kube-api-access-mb6fq:{mountpoint:/var/lib/kubelet/pods/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a/volumes/kubernetes.io~projected/kube-api-access-mb6fq major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/762f4cf8-0912-4663-94e4-ef0c84740817/volumes/kubernetes.io~projected/kube-api-access-n47n8:{mountpoint:/var/lib/kubelet/pods/762f4cf8-0912-4663-94e4-ef0c84740817/volumes/kubernetes.io~projected/kube-api-access-n47n8 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78bd204f-98ed-42b0-9c0d-97baf9aa090d/volumes/kubernetes.io~projected/kube-api-access-fkx26:{mountpoint:/var/lib/kubelet/pods/78bd204f-98ed-42b0-9c0d-97baf9aa090d/volumes/kubernetes.io~projected/kube-api-access-fkx26 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78bd204f-98ed-42b0-9c0d-97baf9aa090d/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/78bd204f-98ed-42b0-9c0d-97baf9aa090d/volumes/kubernetes.io~secret/metrics-certs major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a4f60af-259e-442f-a17b-911beb60fee5/volumes/kubernetes.io~projected/kube-api-access-scqt8:{mountpoint:/var/lib/kubelet/pods/7a4f60af-259e-442f-a17b-911beb60fee5/volumes/kubernetes.io~projected/kube-api-access-scqt8 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81991817-9603-48e1-b25f-0392e58a83b5/volumes/kubernetes.io~projected/kube-api-access-2t95k:{mountpoint:/var/lib/kubelet/pods/81991817-9603-48e1-b25f-0392e58a83b5/volumes/kubernetes.io~projected/kube-api-access-2t95k major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~projected/kube-api-access-4dm7x:{mountpoint:/var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~projected/kube-api-access-4dm7x major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~projected/registry-tls major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b098c99f-60e8-40bd-a431-fa252a476a0b/volumes/kubernetes.io~projected/kube-api-access-nt7hs:{mountpoint:/var/lib/kubelet/pods/b098c99f-60e8-40bd-a431-fa252a476a0b/volumes/kubernetes.io~projected/kube-api-access-nt7hs major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2095b82-50e2-40c5-804e-4f91d84d5e03/volumes/kubernetes.io~projected/kube-api-access-cpl2n:{mountpoint:/var/lib/kubelet/pods/b2095b82-50e2-40c5-804e-4f91d84d5e03/volumes/kubernetes.io~projected/kube-api-access-cpl2n major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2095b82-50e2-40c5-804e-4f91d84d5e03/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b2095b82-50e2-40c5-804e-4f91d84d5e03/volumes/kubernetes.io~secret/serving-cert major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b64026c1-aa80-4092-829e-91e5e8164afe/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/b64026c1-aa80-4092-829e-91e5e8164afe/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~projected/kube-api-access-6qb9d:{mountpoint:/var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~projected/kube-api-access-6qb9d major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~projected/registry-tls major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ceb1e5e7-27de-4c20-9576-52d3989384b7/volumes/kubernetes.io~projected/kube-api-access-n7sjz:{mountpoint:/var/lib/kubelet/pods/ceb1e5e7-27de-4c20-9576-52d3989384b7/volumes/kubernetes.io~projected/kube-api-access-n7sjz major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d15630f9-042e-4c31-8f70-c8733c9dc31e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d15630f9-042e-4c31-8f70-c8733c9dc31e/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df3f7f74-c7a1-4f0d-8328-39f4047b6424/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/df3f7f74-c7a1-4f0d-8328-39f4047b6424/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df3f7f74-c7a1-4f0d-8328-39f4047b6424/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/df3f7f74-c7a1-4f0d-8328-39f4047b6424/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df3f7f74-c7a1-4f0d-8328-39f4047b6424/volumes/kubernetes.io~projected/kube-api-access-ff8l8:{mountpoint:/var/lib/kubelet/pods/df3f7f74-c7a1-4f0d-8328-39f4047b6424/volumes/kubernetes.io~projected/kube-api-access-ff8l8 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2534417-f751-4241-b22d-3ea465dc6a19/volumes/kubernetes.io~projected/kube-api-access-zgql6:{mountpoint:/var/lib/kubelet/pods/e2534417-f751-4241-b22d-3ea465dc6a19/volumes/kubernetes.io~projected/kube-api-access-zgql6 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2534417-f751-4241-b22d-3ea465dc6a19/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e2534417-f751-4241-b22d-3ea465dc6a19/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5b81fe1-bb35-4200-9b69-c49ed020ca64/volumes/kubernetes.io~projected/kube-api-access-7thr9:{mountpoint:/var/lib/kubelet/pods/f5b81fe1-bb35-4200-9b69-c49ed020ca64/volumes/kubernetes.io~projected/kube-api-access-7thr9 major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5b81fe1-bb35-4200-9b69-c49ed020ca64/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/f5b81fe1-bb35-4200-9b69-c49ed020ca64/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f84f08f4-67c2-48b5-b939-ab9269a43bfc/volumes/kubernetes.io~projected/kube-api-access-j7clq:{mountpoint:/var/lib/kubelet/pods/f84f08f4-67c2-48b5-b939-ab9269a43bfc/volumes/kubernetes.io~projected/kube-api-access-j7clq major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f84f08f4-67c2-48b5-b939-ab9269a43bfc/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/f84f08f4-67c2-48b5-b939-ab9269a43bfc/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8fe96da-a52f-4f6c-ae1d-229955f4d9ec/volumes/kubernetes.io~projected/kube-api-access-jh8cn:{mountpoint:/var/lib/kubelet/pods/f8fe96da-a52f-4f6c-ae1d-229955f4d9ec/volumes/kubernetes.io~projected/kube-api-access-jh8cn major:0 minor:329 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4168321282b5fafe5a38ed580d6e0b3fb06f6d4c792074df0ba3b02c1e18bf80/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2fe651adde2ef8e3f83b8e9f216a62c81bc7e8a6fd29ea323b703c00b8af1217/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1c9648461dc812eb842c6f7f02f88e9408a2364f669486574bb167f67f7cda76/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/8d20e63fb2857a1d6c217130e3d80866a7caea4fbe5078cd75ae530bf479d76c/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/64fef12906a393dbf19768171b7f8bfd2ce7575e8f55f157cac507e35d540242/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fb89f7cc73bc755a02e0491d2bae39b18abe7aba409d20fd9dbb7fa225489211/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9825c92fc3051d30ff08a44e54899cbe87a1c15a7fc5acb52cc5cd128d99135d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/80d05f8efb8c71e6d7496c5148102180e447bb3b646e87825285f2595fcd200a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/e44886e7fc1fa952ba8e61218c4b3b1f26a0fc5da2c4c5a06de2151e4b342cb5/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/5af6f744fb6f299da1e667477aeffe44dde11801b13193f55435a03c6b7b4278/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/fdfc64d601844319e8e86827ab7402b062a77e56f84a6060d7b529b2d5cc5229/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/87bbead50d53bf568e95d8ec64c214c9cc814500b3b97dfc7c985163f237f8c0/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a3efa48165ce9bac55bd8f997bcd9aa6fa69292002c95296c79e080eebd89455/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/368835b6e33800a86e5b8ab88c9ce4f9781f04816c358521d6ef90fc73a3755a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/62b05d78cc645d22134ace20dede6d10fe29b1a107c4a2c4d27cd81454d70994/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a7d80a2f593f11ecc4257f07266856a288f0213a2fa2c460beecaece39bfb406/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/f8e24036bb6934a74d16d5cc562fdaf867355cebf7a34b10c68c37cf46f01abb/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/681fa9208b34c9918da2d7fa23e2b0fbe8f683f820aebb530c7512ba2b163840/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a8556e2ae6bca39c7986526177c380bccac7aa4ae49c65b821c93b8d1048b6fd/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e996a06765fd5b1cb509f99719d1ea1d9e8aac7abe45722af1567b34b2e8a52e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/cd479078a8fa2a828fdb9e0399ff55b2afe2abc13b1611b6a34d31058f5c64c2/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/231f773b88a95990dcf59849954956005cd6e5835781b1d08363be486253ed7d/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/22c93666b87a8d90fd9bcbbf1f3d9c7fdde7aee1c14b015c25a97e02e341cb9e/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/abb331b9ca67aea421eaeb02d737f2f64e3d5b47398ee6bf0879dcd90597d43a/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/5d6bddebd72a36fb4e00f79556519f1994256204b944cd7837353277389b1756/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/0527e82bb26b1c42a50c2f5cc8bcb17eddfedf331228999922be2401e6fd873b/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/b6453561873e3e6e38d1727ed32f83303a6c0e72873fdcd9927c83c7ebfd539e/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/070af0776f483b1d2760c54fd829416ec03bd426b2403f91950d8b65db3fabc8/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/f240371c83a51abb24f9b54617e1ac2b5770efc2fa7736d6e5d84a51a7921780/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/b5c30876b2905926ac0e8b905fd9c27b9088cae910be0aa0fe37a052ff6c1d24/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/53ad26ed7dfec805f8aeadcea37ac7d815858fc6cbc8b992d5100d15611edf48/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/1043b03c90aa1b19295689f704cc78b6bdd979589da928427de27caab841746d/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/ff8b9a6662ce5cc0d2df8903fbea0aa3abb14fb8cb59c49e8900142e1736edce/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/99139482955dea7c98324157b0a05b0b1a56455b4c64553c8a4d944a34adf539/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/362980b64ef9b1fabf2c1b55e3e09495b7d96e1f41aca99da7e8244490a0957d/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/80d5925cbdc3faebea604a61e5077ff37fe0ea26e9a2651ffafcd43467bddd7a/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/e98c40b7e663243df98a88ad1e819c03593bce04bfdb6c61bc9ee26aabd18ffd/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/196f3d55aa4250f6fccca3a3264d54d1c9b5153cff1095a1afb2647882e15366/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/b5023406cb5807c771a6f9584c82abfff87bac0b9f621711b6fe30b2be17434e/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/bc307c775bb2b3eeeb8b915f6f23423158dc8f56f5ac8edf15fd64f344cc4fd3/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/7e26cd32eec3d65c6532c475065d156608e90d51f3009db9fd16eb52dcb95a74/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/1d6dd1685f547318d8f768302ca19b4e2b67c9fd8afb8fd68fda79be26130e93/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/90fba637daa6cc6a87e1192eb6683e28f6733ba0887216ceed2c575a694369ed/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/f23e0923f1735953379fc5f3a8159012b52d558d838b7db2dcbcc479b1ad3e0e/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/9b0f7838166dd5b8be2cadae345a5d0e6d7491dde4c7de99e457f86e66b35ea2/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/840d3856b62d8898402edeb90533a93f11855188c3743f7486f387790f47d5b6/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/4b460eba672b8825a2badc0e15b7dec73e682e301a48b6ff5900cc4ba5a672fc/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/b0cd82c8ec515a206239774e20f21f9c8e32b168e11c282b87beb3dcfad0cbe5/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/9b92d0a520bb9cc35b06f066efc62ba3d063384f94f95562587eab080224966c/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/39f13b53e82158702b277ec264d067098f3dcb9c62349b650060be8a147f2359/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/b2755e78b6b9aa15fa485aebfde17cb06c0d24eded59eadfb35a5063d56f5af9/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/43e8336dcd047dd14c85a86dd251a7620afd57caf3163efb33614877b5b31d0a/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/94cc12646a199f67fd8ae3ef1b71636c168f575ebcfa3d5d904453999d3f229e/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/9b5954a852c9792535ab9f262c02424a124099ca1ab87a90c8d93a196b5d4680/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/0d2809f04df55e71116109fdc12ade3553380578ecac952bea5d63fbe2cec380/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/1f81c3d07e3748c8af60fe9848c0b102e4933d3982bd458f001256247874b83a/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/b657079f37fb944ad1acb066abbb76e318694f8b4301167cd9be66017a6299a8/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/978884641b7a8f672ff3b6b6b46d91599890b31844bf2548b4433ba402ea2b16/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/075de60eb8dc78387b0c00d27ae5a91b8d0ea6a1cb4e6425d56297cbfc02fcd8/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/5ff96b058b20c9fb3505e1e00180635de496e7754d9799ad373c3408fe49279f/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-433:{mountpoint:/var/lib/containers/storage/overlay/475179e685c08c3b33f5d6c2387915d54a03fb7a53d645ff22550eea90d5b516/merged major:0 minor:433 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/2dfbc663413612e3f4ffd40cdfbced4bf08a20a712c3d0e3fa3f1193fabcbeb6/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/4dd6e457add63c7808c99504c4ce1fa42133c01dda92772603ffaa6d20b1a1f8/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/a7aa79d1bac301804812ba0d590bb25ec89fe824ea8b8743be3efb769b9a8504/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/4b8238b12dc9696881ab240c19860dcc2d1a4093a81c935c2da5808e9393a4b0/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/463ac752328489a71d0a3d7bcc21e47888c37796fa63b7f2cf534b1d92c5a924/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-463:{mountpoint:/var/lib/containers/storage/overlay/6588f3d0b0a3ae941771fa6f1a2f5174269f941c8513365af2d804ca6936ef0c/merged major:0 minor:463 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/abe8939bbed77af8c5068b87b4260a6845fdcb356dea913b8664a82effb428ff/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/36d2b1913552fb756aef4ddc489ce340f8019cee4f3ce9c108d310873e54e196/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/76c65612950562284c9ab11913aa2a0d5444e8a8a97217316297be2a6746b5ee/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/8f3cf0d6a742c7c1b7c44c5980429b6a408012197f56e2a6216207eaee661ce7/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/b2cafe168cefd9fb1a0b65a07e27407fef94fcad7584ffc3d57fe94921fb5462/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/d59074ef91188e2075c67f83232fcd84945f281073c4858d1798d8a070c73d75/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/61e4d17fae54190afe41e30acd6ed76fd57912fe9e9619e714cd7360d7e536c7/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/dd80a8be28c8fa2b2061e531ba88b533c7a079d1d9cbd4abd562984ccc35af5a/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/0cda640d419bfdd3f74a3fe5fed245e64f7770c399bc7f3901f3ece900e75a10/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/2a12279f8dc3f44dba4586b88a7ced0c75a910cb2370150b93686a176bfb4fa1/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/39f242bf7f12de0bc7002f8da0ac692f4d59aff0281178f3d01ae56797bd6fd4/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/53252f9a2c56ae72ede72203c3398d347c38c2430b8b9c38d40d90a2c5fc4a31/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 00:57:34.450535 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.450049 5786 manager.go:217] Machine: {Timestamp:2026-11-04 00:57:34.449555926 +0000 UTC m=+0.065901683 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099045 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23bb6f8d874a9138b8d9423342e218 SystemUUID:ec23bb6f-8d87-4a91-38b8-d9423342e218 BootID:d3c05ee4-d1de-42e4-8666-820d5fc407cc Filesystems:[{Device:/run/containers/storage/overlay-containers/2189e299abfb14d8bc685d7a52518f27aaf275387ddbc8cacafd793743f4f5ef/userdata/shm DeviceMajor:0 DeviceMinor:381 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e2534417-f751-4241-b22d-3ea465dc6a19/volumes/kubernetes.io~projected/kube-api-access-zgql6 DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6239dd443759994df4c3cd47ab57a764dba24aed318f6e9c06b20f08f7a90ebb/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa76b32e7ca58dbeb41e98e9dc715191e66fa15a184fc75e80230e779914d124/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-463 DeviceMajor:0 DeviceMinor:463 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3bd0fcf4-1163-444a-95cd-be27096da34d/volumes/kubernetes.io~projected/kube-api-access-q44f4 DeviceMajor:0 DeviceMinor:243 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49b06b427edd7f210d0b64acff80f9410d003f962b4e1dc21585105e8b31c7e3/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/576edbd7-09bc-44d8-ba89-12af92909132/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc001fd8e0377241c1f20264c489dbba015c928138f744f48deb41a1accad95c/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0aa691c57a3767004d22bad031b6714c9202828dbc8ac4b0c4a9136a8bf024b2/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a2d786d3267c9a6dfe3c4e55d983cc405632b2f97a5ebcbb6188ca0af23333a/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b64026c1-aa80-4092-829e-91e5e8164afe/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:374 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-433 DeviceMajor:0 DeviceMinor:433 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d15630f9-042e-4c31-8f70-c8733c9dc31e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c454bd79a3e2867138ba42d7a6bd571d65d85986eca640be0d9902327f1ee9e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3183dfa600ae4dc32d285f2fa478fba8384bc67ec4e8397aace58f7e1f90a6d3/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f84f08f4-67c2-48b5-b939-ab9269a43bfc/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:369 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78bd204f-98ed-42b0-9c0d-97baf9aa090d/volumes/kubernetes.io~projected/kube-api-access-fkx26 DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ea2674aeefe26451bfec13e94671d6b93ea1ddb36e50af31affbb530e658065/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fd08dc37586bbaccc3bc48aeaf69b34346ad767e38bae7882ea89edbe1f7563/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f84f08f4-67c2-48b5-b939-ab9269a43bfc/volumes/kubernetes.io~projected/kube-api-access-j7clq DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fa177a8-889a-4a1e-9616-7d2f81a55c8b/volumes/kubernetes.io~projected/kube-api-access-g8s82 DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77ec6df6a03a3682953ac1001b9b484abf1f9f75d62f158da113f8245237946f/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45ffe87d163ac212b283931208beddac26db1866277950b6880a9c94be72cc14/userdata/shm DeviceMajor:0 DeviceMinor:379 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e8768c7ee68de04c9ab28e87b723cbbce6e1c7ea8ba8d0bc205daa3f02c1c8c/userdata/shm DeviceMajor:0 DeviceMinor:425 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2a5a1afd44cdd3403fe3b2f1346ef9fe83fb402a511e722f18289c06bac754f/userdata/shm DeviceMajor:0 DeviceMinor:427 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2095b82-50e2-40c5-804e-4f91d84d5e03/volumes/kubernetes.io~projected/kube-api-access-cpl2n DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f5b81fe1-bb35-4200-9b69-c49ed020ca64/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:367 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6df800a919c45b98191e8c8a9d0256d8a7e8bacfe6c5ec83ec3f43ba56b3ea0d/userdata/shm DeviceMajor:0 DeviceMinor:385 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f8fe96da-a52f-4f6c-ae1d-229955f4d9ec/volumes/kubernetes.io~projected/kube-api-access-jh8cn DeviceMajor:0 DeviceMinor:329 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:368 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df3f7f74-c7a1-4f0d-8328-39f4047b6424/volumes/kubernetes.io~projected/kube-api-access-ff8l8 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e/volumes/kubernetes.io~projected/kube-api-access-5wtnl DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7a4f60af-259e-442f-a17b-911beb60fee5/volumes/kubernetes.io~projected/kube-api-access-scqt8 DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df3f7f74-c7a1-4f0d-8328-39f4047b6424/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b098c99f-60e8-40bd-a431-fa252a476a0b/volumes/kubernetes.io~projected/kube-api-access-nt7hs DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21952e92-3601-4969-adbd-6c6ff9b87074/volumes/kubernetes.io~projected/kube-api-access-2b9md DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2248ed8a-1d14-4a43-8e03-7bf83ed903a6/volumes/kubernetes.io~projected/kube-api-access-gq6kt DeviceMajor:0 DeviceMinor:242 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92fe1d699adb2132995a76d9f46052f682f1321c3b82552bcfae33f4bebc2886/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abdcb4619f16a6dd66cf8370e60cfcbd1618f909b34cad8d6dd00a0ea97b6691/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3775001e4a4db27460ffabfcefcfb3ba324e51b0a0a952ae48ea5a258c0e45da/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/593868da-9895-4d98-9da1-4fa815c11674/volumes/kubernetes.io~projected/kube-api-access-psm57 DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c89163b-5d76-4794-93bb-56840ed333d7/volumes/kubernetes.io~projected/kube-api-access-jc65g DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/762f4cf8-0912-4663-94e4-ef0c84740817/volumes/kubernetes.io~projected/kube-api-access-n47n8 DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f5f2f4cd50771092c6b4f72faa2f4f864c51027da8fe854402d62837b1d4f1d/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f5b81fe1-bb35-4200-9b69-c49ed020ca64/volumes/kubernetes.io~projected/kube-api-access-7thr9 DeviceMajor:0 DeviceMinor:338 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/674fcbdbd6d5767b57e4719770ebe1b2a124ae6ccc3463fefe335beaa79a91d4/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/509fdb2ef10fc63c144f71b866b04e9f5ac3a494b8fe14e343da6cabdcd96233/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68909b0c-2aad-4e3f-ab79-d13d21561db5/volumes/kubernetes.io~projected/kube-api-access-bvl2j DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~projected/kube-api-access-6qb9d DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/ceb1e5e7-27de-4c20-9576-52d3989384b7/volumes/kubernetes.io~projected/kube-api-access-n7sjz DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3181e3c4-6f85-4302-965b-672bbf7534ba/volumes/kubernetes.io~projected/kube-api-access-6zkfp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/576edbd7-09bc-44d8-ba89-12af92909132/volumes/kubernetes.io~projected/kube-api-access-74c2k DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f37ebc5fcf7f274bdf12d6e3b5ea789795044ce760ffa663c6f5f5a54e681ff/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94b25173f5fcdf9544a823009ff9682def60a2f6ad23e1e41e2bdb9644325959/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f2e5d44e6a63a5db33529802c38926684727b226ee5d611619d8cace6df161c/userdata/shm DeviceMajor:0 DeviceMinor:377 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ad9dfd7-097e-4491-a3b3-ba3c89a52992/volumes/kubernetes.io~projected/kube-api-access-h4p4d DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6cd2148f8da7e3f2640a201e9a468c6b9592a186ac5ec4a477b4332b204f25f/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b2095b82-50e2-40c5-804e-4f91d84d5e03/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a/volumes/kubernetes.io~projected/kube-api-access-mb6fq DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/011d3a1a-a2a5-4ef3-a871-453fb22817d5/volumes/kubernetes.io~projected/kube-api-access-jljcp DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e2534417-f751-4241-b22d-3ea465dc6a19/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:370 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f72adec94ce451d7d63fab7300af4f04d09c96e9f9eab63f87f971a3b599a95/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ffc847eeba4c623cb9e5255c1d81b432fad4ff7638531bdd0db0112025965b5/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/volumes/kubernetes.io~projected/kube-api-access-llcg6 DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~projected/kube-api-access-4dm7x DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fe0e753a5e840cbc2828ee923374bdc8eb3f0e97ca6752d940d2fd231ce615d/userdata/shm DeviceMajor:0 DeviceMinor:383 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae166c7a149d538c0a9bbd6f65b27c277ccf8d39ef6e3b16efed217fbc5fb74a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0ebe7f81-60bb-423b-a871-c2bab48db5c3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:424 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df3f7f74-c7a1-4f0d-8328-39f4047b6424/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/68909b0c-2aad-4e3f-ab79-d13d21561db5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bd0fcf4-1163-444a-95cd-be27096da34d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:373 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/81991817-9603-48e1-b25f-0392e58a83b5/volumes/kubernetes.io~projected/kube-api-access-2t95k DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21952e92-3601-4969-adbd-6c6ff9b87074/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fce1cf781706c1c7811cbb7d7e8cb8d1afddefbdd292e8462c5c13f5b13645c2/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78bd204f-98ed-42b0-9c0d-97baf9aa090d/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:423 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac1b456d6d1432d94931f8726a3c15cdbde7f3f8e2c8199bb875232786d24fe6/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1fe0e753a5e840c MacAddress:ba:c5:3f:46:99:76 Speed:10000 Mtu:8901} {Name:2189e299abfb14d MacAddress:a2:a0:b3:35:f3:55 Speed:10000 Mtu:8901} {Name:2a2d786d3267c9a MacAddress:96:2c:14:e6:f1:92 Speed:10000 Mtu:8901} {Name:2fd08dc37586bba MacAddress:ba:45:e4:c0:9b:cb Speed:10000 Mtu:8901} {Name:3183dfa600ae4dc MacAddress:06:24:40:ba:68:8f Speed:10000 Mtu:8901} {Name:3e8768c7ee68de0 MacAddress:e2:4d:a3:29:ff:9b Speed:10000 Mtu:8901} {Name:45ffe87d163ac21 MacAddress:ce:c2:d4:54:85:92 Speed:10000 Mtu:8901} {Name:4f2e5d44e6a63a5 MacAddress:22:01:3b:b2:6a:bf Speed:10000 Mtu:8901} {Name:4f37ebc5fcf7f27 MacAddress:e2:7a:f8:75:7e:c8 Speed:10000 Mtu:8901} {Name:4f5f2f4cd507710 MacAddress:42:f4:a9:e8:a5:70 Speed:10000 Mtu:8901} {Name:509fdb2ef10fc63 MacAddress:c2:75:f9:ec:d7:c7 Speed:10000 Mtu:8901} {Name:6239dd443759994 MacAddress:ee:5c:51:6d:78:0a Speed:10000 Mtu:8901} {Name:674fcbdbd6d5767 MacAddress:3e:a8:9d:7d:27:27 Speed:10000 Mtu:8901} {Name:6df800a919c45b9 MacAddress:3e:94:a2:a9:5b:b1 Speed:10000 Mtu:8901} {Name:77ec6df6a03a368 MacAddress:56:f7:40:d4:9f:10 Speed:10000 Mtu:8901} {Name:92fe1d699adb213 MacAddress:0e:71:65:69:e9:e7 Speed:10000 Mtu:8901} {Name:94b25173f5fcdf9 MacAddress:d2:5c:d5:a2:4c:a2 Speed:10000 Mtu:8901} {Name:aa76b32e7ca58db MacAddress:56:61:60:c6:86:48 Speed:10000 Mtu:8901} {Name:ac1b456d6d1432d MacAddress:5a:1d:64:3f:cb:d6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d3:84:3f:43 Speed:0 Mtu:9001} {Name:br-int MacAddress:22:f2:ab:bb:16:aa Speed:0 Mtu:8901} {Name:c2a5a1afd44cdd3 MacAddress:3a:d0:c8:d3:68:73 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d3:84:3f:43 Speed:0 Mtu:9001} {Name:fc001fd8e037724 MacAddress:32:ca:8a:a8:90:eb Speed:10000 Mtu:8901} {Name:fce1cf781706c1c MacAddress:86:fd:c3:6a:ff:1d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:5a:82:81:8e:5d:bd Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:f6:c8:11:5e:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 00:57:34.450861 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.450539 5786 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 00:57:34.450861 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.450591 5786 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 00:57:34.450982 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.450952 5786 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 00:57:34.451159 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.450985 5786 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-94.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 00:57:34.451250 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.451163 5786 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 00:57:34.451250 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.451176 5786 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 00:57:34.451250 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.451192 5786 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:57:34.451250 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.451215 5786 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:57:34.451250 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.451243 5786 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 00:57:34.451480 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.451468 5786 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 00:57:34.451609 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.451596 5786 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 00:57:34.451670 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.451645 5786 kubelet.go:493] "Attempting to sync node with API server" Nov 04 00:57:34.451670 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.451666 5786 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 00:57:34.451759 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.451681 5786 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 00:57:34.451759 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.451694 5786 kubelet.go:405] "Adding apiserver pod source" Nov 04 00:57:34.451759 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.451706 5786 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 00:57:34.452455 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.452438 5786 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 00:57:34.452455 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.452452 5786 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:57:34.452979 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.452961 5786 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 00:57:34.455777 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.455764 5786 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 00:57:34.455823 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.455778 5786 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:57:34.455898 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.455888 5786 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 00:57:34.456084 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456073 5786 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 00:57:34.456113 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456103 5786 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 00:57:34.456213 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456204 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 00:57:34.456248 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456218 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 00:57:34.456248 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456224 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 00:57:34.456248 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456230 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 00:57:34.456248 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456237 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 00:57:34.456248 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456243 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 00:57:34.456248 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456248 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 00:57:34.456400 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456254 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 00:57:34.456400 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456261 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 00:57:34.456400 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456267 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 00:57:34.456400 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456290 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 00:57:34.456400 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456300 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 00:57:34.456400 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456333 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 00:57:34.456400 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.456339 5786 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 00:57:34.459804 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.459791 5786 server.go:1263] "Started kubelet" Nov 04 00:57:34.460211 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.459874 5786 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 00:57:34.460211 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.459888 5786 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 00:57:34.460211 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.459966 5786 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 00:57:34.460387 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.460331 5786 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 00:57:34.460419 ip-10-0-136-94 systemd[1]: Started Kubernetes Kubelet. Nov 04 00:57:34.464738 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.463517 5786 server.go:317] "Adding debug handlers to kubelet server" Nov 04 00:57:34.467171 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.466163 5786 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 00:57:34.467171 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.466276 5786 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 00:57:34.467171 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.466353 5786 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:51:22 +0000 UTC" deadline="2028-04-26 14:51:44.362347803 +0000 UTC" Nov 04 00:57:34.467171 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.466398 5786 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12949h54m9.89595534s" Nov 04 00:57:34.467171 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.466498 5786 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 00:57:34.467171 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.466512 5786 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 00:57:34.467171 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:57:34.466615 5786 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-94.us-west-2.compute.internal\" not found" Nov 04 00:57:34.467537 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.467236 5786 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 00:57:34.476489 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.476469 5786 factory.go:55] Registering systemd factory Nov 04 00:57:34.476489 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.476491 5786 factory.go:223] Registration of the systemd container factory successfully Nov 04 00:57:34.477972 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.476778 5786 factory.go:153] Registering CRI-O factory Nov 04 00:57:34.477972 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.476793 5786 factory.go:223] Registration of the crio container factory successfully Nov 04 00:57:34.477972 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.476863 5786 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 00:57:34.477972 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.476884 5786 factory.go:103] Registering Raw factory Nov 04 00:57:34.477972 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.476901 5786 manager.go:1199] Started watching for new ooms in manager Nov 04 00:57:34.477972 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.477268 5786 manager.go:319] Starting recovery of all containers Nov 04 00:57:34.479694 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479651 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" volumeName="kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479697 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5b81fe1-bb35-4200-9b69-c49ed020ca64" volumeName="kubernetes.io/projected/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-api-access-7thr9" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479713 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d69ef80-a3b8-4c02-a636-6b30e21e5b2a" volumeName="kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-env-overrides" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479727 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac49addd-a050-4d12-bdc0-20345e446a15" volumeName="kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-installation-pull-secrets" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479740 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b098c99f-60e8-40bd-a431-fa252a476a0b" volumeName="kubernetes.io/projected/b098c99f-60e8-40bd-a431-fa252a476a0b-kube-api-access-nt7hs" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479752 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbd1d155-f232-4679-a70a-ac3911aeb391" volumeName="kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-bound-sa-token" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479764 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbd1d155-f232-4679-a70a-ac3911aeb391" volumeName="kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-installation-pull-secrets" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479787 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceb1e5e7-27de-4c20-9576-52d3989384b7" volumeName="kubernetes.io/configmap/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-daemon-config" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479801 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df3f7f74-c7a1-4f0d-8328-39f4047b6424" volumeName="kubernetes.io/projected/df3f7f74-c7a1-4f0d-8328-39f4047b6424-kube-api-access-ff8l8" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479810 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84f08f4-67c2-48b5-b939-ab9269a43bfc" volumeName="kubernetes.io/projected/f84f08f4-67c2-48b5-b939-ab9269a43bfc-kube-api-access-j7clq" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479818 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" volumeName="kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cni-binary-copy" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479825 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3181e3c4-6f85-4302-965b-672bbf7534ba" volumeName="kubernetes.io/configmap/3181e3c4-6f85-4302-965b-672bbf7534ba-iptables-alerter-script" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479835 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bd0fcf4-1163-444a-95cd-be27096da34d" volumeName="kubernetes.io/empty-dir/3bd0fcf4-1163-444a-95cd-be27096da34d-tmp-dir" seLinuxMountContext="" Nov 04 00:57:34.479853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479847 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac49addd-a050-4d12-bdc0-20345e446a15" volumeName="kubernetes.io/empty-dir/ac49addd-a050-4d12-bdc0-20345e446a15-ca-trust-extracted" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479864 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbd1d155-f232-4679-a70a-ac3911aeb391" volumeName="kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-kube-api-access-6qb9d" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479878 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2534417-f751-4241-b22d-3ea465dc6a19" volumeName="kubernetes.io/configmap/e2534417-f751-4241-b22d-3ea465dc6a19-config" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479892 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2534417-f751-4241-b22d-3ea465dc6a19" volumeName="kubernetes.io/secret/e2534417-f751-4241-b22d-3ea465dc6a19-serving-cert" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479904 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84f08f4-67c2-48b5-b939-ab9269a43bfc" volumeName="kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479917 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21952e92-3601-4969-adbd-6c6ff9b87074" volumeName="kubernetes.io/configmap/21952e92-3601-4969-adbd-6c6ff9b87074-telemetry-config" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479930 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bd0fcf4-1163-444a-95cd-be27096da34d" volumeName="kubernetes.io/configmap/3bd0fcf4-1163-444a-95cd-be27096da34d-config-volume" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479945 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68909b0c-2aad-4e3f-ab79-d13d21561db5" volumeName="kubernetes.io/configmap/68909b0c-2aad-4e3f-ab79-d13d21561db5-service-ca-bundle" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479958 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68909b0c-2aad-4e3f-ab79-d13d21561db5" volumeName="kubernetes.io/secret/68909b0c-2aad-4e3f-ab79-d13d21561db5-serving-cert" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.479970 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d15630f9-042e-4c31-8f70-c8733c9dc31e" volumeName="kubernetes.io/secret/d15630f9-042e-4c31-8f70-c8733c9dc31e-agent-certs" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480017 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28aa6fa1-33f8-450e-a1f1-d382e7cf6a47" volumeName="kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-stats-auth" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480032 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c89163b-5d76-4794-93bb-56840ed333d7" volumeName="kubernetes.io/projected/2c89163b-5d76-4794-93bb-56840ed333d7-kube-api-access-jc65g" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480046 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a4f60af-259e-442f-a17b-911beb60fee5" volumeName="kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480066 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbd1d155-f232-4679-a70a-ac3911aeb391" volumeName="kubernetes.io/empty-dir/cbd1d155-f232-4679-a70a-ac3911aeb391-ca-trust-extracted" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480080 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="011d3a1a-a2a5-4ef3-a871-453fb22817d5" volumeName="kubernetes.io/empty-dir/011d3a1a-a2a5-4ef3-a871-453fb22817d5-tmp-dir" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480100 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28aa6fa1-33f8-450e-a1f1-d382e7cf6a47" volumeName="kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480114 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e" volumeName="kubernetes.io/secret/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e-cert" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480154 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="576edbd7-09bc-44d8-ba89-12af92909132" volumeName="kubernetes.io/configmap/576edbd7-09bc-44d8-ba89-12af92909132-config" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480168 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac49addd-a050-4d12-bdc0-20345e446a15" volumeName="kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-bound-sa-token" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480182 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbd1d155-f232-4679-a70a-ac3911aeb391" volumeName="kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-certificates" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480196 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df3f7f74-c7a1-4f0d-8328-39f4047b6424" volumeName="kubernetes.io/empty-dir/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-tuned" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480208 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2534417-f751-4241-b22d-3ea465dc6a19" volumeName="kubernetes.io/projected/e2534417-f751-4241-b22d-3ea465dc6a19-kube-api-access-zgql6" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480222 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d15630f9-042e-4c31-8f70-c8733c9dc31e" volumeName="kubernetes.io/configmap/d15630f9-042e-4c31-8f70-c8733c9dc31e-konnectivity-ca" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480235 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21952e92-3601-4969-adbd-6c6ff9b87074" volumeName="kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480247 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2248ed8a-1d14-4a43-8e03-7bf83ed903a6" volumeName="kubernetes.io/projected/2248ed8a-1d14-4a43-8e03-7bf83ed903a6-kube-api-access-gq6kt" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480262 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28aa6fa1-33f8-450e-a1f1-d382e7cf6a47" volumeName="kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480275 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fa177a8-889a-4a1e-9616-7d2f81a55c8b" volumeName="kubernetes.io/projected/2fa177a8-889a-4a1e-9616-7d2f81a55c8b-kube-api-access-g8s82" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480289 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d69ef80-a3b8-4c02-a636-6b30e21e5b2a" volumeName="kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovnkube-config" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480303 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac49addd-a050-4d12-bdc0-20345e446a15" volumeName="kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480316 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac49addd-a050-4d12-bdc0-20345e446a15" volumeName="kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-image-registry-private-configuration" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480329 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbd1d155-f232-4679-a70a-ac3911aeb391" volumeName="kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-trusted-ca" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480345 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ebe7f81-60bb-423b-a871-c2bab48db5c3" volumeName="kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480359 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d69ef80-a3b8-4c02-a636-6b30e21e5b2a" volumeName="kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovnkube-script-lib" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480372 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="762f4cf8-0912-4663-94e4-ef0c84740817" volumeName="kubernetes.io/configmap/762f4cf8-0912-4663-94e4-ef0c84740817-serviceca" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480385 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac49addd-a050-4d12-bdc0-20345e446a15" volumeName="kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-trusted-ca" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480398 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b64026c1-aa80-4092-829e-91e5e8164afe" volumeName="kubernetes.io/configmap/b64026c1-aa80-4092-829e-91e5e8164afe-nginx-conf" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480411 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceb1e5e7-27de-4c20-9576-52d3989384b7" volumeName="kubernetes.io/configmap/ceb1e5e7-27de-4c20-9576-52d3989384b7-cni-binary-copy" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480427 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5b81fe1-bb35-4200-9b69-c49ed020ca64" volumeName="kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480440 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e" volumeName="kubernetes.io/projected/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e-kube-api-access-5wtnl" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480454 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68909b0c-2aad-4e3f-ab79-d13d21561db5" volumeName="kubernetes.io/configmap/68909b0c-2aad-4e3f-ab79-d13d21561db5-trusted-ca-bundle" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480467 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df3f7f74-c7a1-4f0d-8328-39f4047b6424" volumeName="kubernetes.io/empty-dir/df3f7f74-c7a1-4f0d-8328-39f4047b6424-tmp" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480480 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2534417-f751-4241-b22d-3ea465dc6a19" volumeName="kubernetes.io/configmap/e2534417-f751-4241-b22d-3ea465dc6a19-trusted-ca" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480493 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8fe96da-a52f-4f6c-ae1d-229955f4d9ec" volumeName="kubernetes.io/projected/f8fe96da-a52f-4f6c-ae1d-229955f4d9ec-kube-api-access-jh8cn" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480536 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21952e92-3601-4969-adbd-6c6ff9b87074" volumeName="kubernetes.io/projected/21952e92-3601-4969-adbd-6c6ff9b87074-kube-api-access-2b9md" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480550 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" volumeName="kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480563 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bd0fcf4-1163-444a-95cd-be27096da34d" volumeName="kubernetes.io/projected/3bd0fcf4-1163-444a-95cd-be27096da34d-kube-api-access-q44f4" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480590 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="762f4cf8-0912-4663-94e4-ef0c84740817" volumeName="kubernetes.io/projected/762f4cf8-0912-4663-94e4-ef0c84740817-kube-api-access-n47n8" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480603 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac49addd-a050-4d12-bdc0-20345e446a15" volumeName="kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-registry-certificates" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480618 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbd1d155-f232-4679-a70a-ac3911aeb391" volumeName="kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480632 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28aa6fa1-33f8-450e-a1f1-d382e7cf6a47" volumeName="kubernetes.io/projected/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-kube-api-access-llcg6" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480645 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3181e3c4-6f85-4302-965b-672bbf7534ba" volumeName="kubernetes.io/projected/3181e3c4-6f85-4302-965b-672bbf7534ba-kube-api-access-6zkfp" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480658 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bd0fcf4-1163-444a-95cd-be27096da34d" volumeName="kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480671 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="576edbd7-09bc-44d8-ba89-12af92909132" volumeName="kubernetes.io/secret/576edbd7-09bc-44d8-ba89-12af92909132-serving-cert" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480683 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="593868da-9895-4d98-9da1-4fa815c11674" volumeName="kubernetes.io/projected/593868da-9895-4d98-9da1-4fa815c11674-kube-api-access-psm57" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480691 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68909b0c-2aad-4e3f-ab79-d13d21561db5" volumeName="kubernetes.io/empty-dir/68909b0c-2aad-4e3f-ab79-d13d21561db5-tmp" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480699 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84f08f4-67c2-48b5-b939-ab9269a43bfc" volumeName="kubernetes.io/configmap/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-config" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480707 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28aa6fa1-33f8-450e-a1f1-d382e7cf6a47" volumeName="kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-default-certificate" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480715 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d69ef80-a3b8-4c02-a636-6b30e21e5b2a" volumeName="kubernetes.io/projected/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-kube-api-access-mb6fq" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480722 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81991817-9603-48e1-b25f-0392e58a83b5" volumeName="kubernetes.io/projected/81991817-9603-48e1-b25f-0392e58a83b5-kube-api-access-2t95k" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480733 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2095b82-50e2-40c5-804e-4f91d84d5e03" volumeName="kubernetes.io/projected/b2095b82-50e2-40c5-804e-4f91d84d5e03-kube-api-access-cpl2n" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480744 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" volumeName="kubernetes.io/projected/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-kube-api-access-h4p4d" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480755 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d69ef80-a3b8-4c02-a636-6b30e21e5b2a" volumeName="kubernetes.io/secret/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480763 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b64026c1-aa80-4092-829e-91e5e8164afe" volumeName="kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480771 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5b81fe1-bb35-4200-9b69-c49ed020ca64" volumeName="kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480779 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5b81fe1-bb35-4200-9b69-c49ed020ca64" volumeName="kubernetes.io/empty-dir/f5b81fe1-bb35-4200-9b69-c49ed020ca64-data-volume" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480786 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="576edbd7-09bc-44d8-ba89-12af92909132" volumeName="kubernetes.io/projected/576edbd7-09bc-44d8-ba89-12af92909132-kube-api-access-74c2k" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480794 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78bd204f-98ed-42b0-9c0d-97baf9aa090d" volumeName="kubernetes.io/projected/78bd204f-98ed-42b0-9c0d-97baf9aa090d-kube-api-access-fkx26" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480803 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78bd204f-98ed-42b0-9c0d-97baf9aa090d" volumeName="kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480810 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2095b82-50e2-40c5-804e-4f91d84d5e03" volumeName="kubernetes.io/configmap/b2095b82-50e2-40c5-804e-4f91d84d5e03-config" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480817 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceb1e5e7-27de-4c20-9576-52d3989384b7" volumeName="kubernetes.io/projected/ceb1e5e7-27de-4c20-9576-52d3989384b7-kube-api-access-n7sjz" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480825 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="011d3a1a-a2a5-4ef3-a871-453fb22817d5" volumeName="kubernetes.io/projected/011d3a1a-a2a5-4ef3-a871-453fb22817d5-kube-api-access-jljcp" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480833 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68909b0c-2aad-4e3f-ab79-d13d21561db5" volumeName="kubernetes.io/empty-dir/68909b0c-2aad-4e3f-ab79-d13d21561db5-snapshots" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480841 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68909b0c-2aad-4e3f-ab79-d13d21561db5" volumeName="kubernetes.io/projected/68909b0c-2aad-4e3f-ab79-d13d21561db5-kube-api-access-bvl2j" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480849 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac49addd-a050-4d12-bdc0-20345e446a15" volumeName="kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-kube-api-access-4dm7x" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480856 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2095b82-50e2-40c5-804e-4f91d84d5e03" volumeName="kubernetes.io/secret/b2095b82-50e2-40c5-804e-4f91d84d5e03-serving-cert" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480864 5786 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbd1d155-f232-4679-a70a-ac3911aeb391" volumeName="kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-image-registry-private-configuration" seLinuxMountContext="" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480880 5786 reconstruct.go:97] "Volume reconstruction finished" Nov 04 00:57:34.482610 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.480885 5786 reconciler.go:29] "Reconciler: start to sync state" Nov 04 00:57:34.486287 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.483480 5786 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 00:57:34.492423 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.492144 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 00:57:34.494353 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.494331 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 00:57:34.497462 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.497433 5786 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 00:57:34.497547 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.497464 5786 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 00:57:34.497547 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.497490 5786 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 00:57:34.497665 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:57:34.497537 5786 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 00:57:34.512018 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.511976 5786 generic.go:356] "Generic (PLEG): container finished" podID="f7d4196ef8a29ed098777bd3e813be05" containerID="14d084ee274713bc6669fe17c4f3e4d88b1c22eefda56065fa0103aa3f47986a" exitCode=0 Nov 04 00:57:34.515850 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.515823 5786 generic.go:356] "Generic (PLEG): container finished" podID="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" containerID="55a3767e770d6ada80fa86d4c0a5d999b212513ee0fc6bb6e613ea343a746607" exitCode=0 Nov 04 00:57:34.515850 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.515846 5786 generic.go:356] "Generic (PLEG): container finished" podID="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" containerID="e9ef453ab5ba52ef95f915b1492cb8ea249e8a85aa9e6402e4f2888e00011e52" exitCode=0 Nov 04 00:57:34.515850 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.515856 5786 generic.go:356] "Generic (PLEG): container finished" podID="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" containerID="ff578754400e935269b50ebea664bcd577f265a756e630ed6ab15e57cb54dd1b" exitCode=0 Nov 04 00:57:34.516077 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.515864 5786 generic.go:356] "Generic (PLEG): container finished" podID="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" containerID="151f3979c52025e853d0042ed81400d4b3e5539a7ffdd86d1314b93953ea8ca7" exitCode=0 Nov 04 00:57:34.516077 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.515869 5786 generic.go:356] "Generic (PLEG): container finished" podID="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" containerID="de9469ad3c5350ca9149c58248c31388d455dc9ff505691bc31055471b4cf7b1" exitCode=0 Nov 04 00:57:34.516077 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.515874 5786 generic.go:356] "Generic (PLEG): container finished" podID="2ad9dfd7-097e-4491-a3b3-ba3c89a52992" containerID="b6e690f3991b135b62923eba6b1f5333501b598d8aaf628443efa31bdb945cdc" exitCode=0 Nov 04 00:57:34.548636 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.548564 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 00:57:34.548636 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.548597 5786 generic.go:356] "Generic (PLEG): container finished" podID="e2534417-f751-4241-b22d-3ea465dc6a19" containerID="950227cc5d7e90c9d8958582b2659aeccb3e10e1eb241adbb3125cc62fba7272" exitCode=255 Nov 04 00:57:34.566701 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:57:34.566675 5786 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-94.us-west-2.compute.internal\" not found" Nov 04 00:57:34.592361 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.592333 5786 manager.go:324] Recovery completed Nov 04 00:57:34.597639 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:57:34.597612 5786 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 04 00:57:34.614955 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.614932 5786 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:34.617024 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.616999 5786 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:34.617133 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.617031 5786 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:34.617133 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.617043 5786 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:34.619955 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.619935 5786 cpu_manager.go:226] "Starting" policy="none" Nov 04 00:57:34.619955 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.619951 5786 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 00:57:34.620108 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.619955 5786 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:34.620108 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.619972 5786 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 00:57:34.620235 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.620148 5786 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 00:57:34.620235 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.620165 5786 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 00:57:34.620235 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.620191 5786 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 00:57:34.620235 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.620201 5786 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 00:57:34.620235 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.620209 5786 policy_none.go:50] "Start" Nov 04 00:57:34.620235 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.620227 5786 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 00:57:34.620497 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.620242 5786 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 00:57:34.620497 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.620426 5786 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 00:57:34.620497 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.620437 5786 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 00:57:34.620497 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.620445 5786 policy_none.go:44] "Start" Nov 04 00:57:34.620985 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.620968 5786 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:34.621061 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.620996 5786 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:34.621061 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.621010 5786 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:34.625300 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.625274 5786 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 00:57:34.627856 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.627841 5786 manager.go:341] "Starting Device Plugin manager" Nov 04 00:57:34.627941 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:57:34.627868 5786 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 00:57:34.627941 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.627877 5786 server.go:87] "Starting device plugin registration server" Nov 04 00:57:34.628142 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.628130 5786 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 00:57:34.628201 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.628147 5786 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 00:57:34.628263 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.628239 5786 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 00:57:34.628514 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.628345 5786 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 00:57:34.628514 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.628396 5786 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 00:57:34.628514 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.628403 5786 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 00:57:34.628640 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.628480 5786 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T00:57:34.628381332Z","UUID":"b5d7de89-6f12-4e2b-b090-de59a78ae089","Handler":null,"Name":"","Endpoint":""} Nov 04 00:57:34.630425 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:57:34.630401 5786 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 00:57:34.630517 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:57:34.630440 5786 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-94.us-west-2.compute.internal\" not found" Nov 04 00:57:34.630690 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.630674 5786 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 00:57:34.630745 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.630716 5786 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 00:57:34.653853 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.653828 5786 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:34.653982 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.653922 5786 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 00:57:34.655287 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.655259 5786 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:34.655458 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.655442 5786 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:34.703748 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.703715 5786 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:34.730632 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.730604 5786 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:34.731852 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.731828 5786 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:34.731942 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.731862 5786 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:34.731942 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.731876 5786 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-94.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:34.731942 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.731931 5786 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:34.798418 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.798353 5786 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal"] Nov 04 00:57:34.799103 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.799026 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" event={"ID":"f7d4196ef8a29ed098777bd3e813be05","Type":"ContainerStarted","Data":"e2289426ba9bce29e31ce78928c58b9b000e218410a76ed5596cab242084edf9"} Nov 04 00:57:34.799202 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.799114 5786 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:34.799202 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.799144 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" event={"ID":"f7d4196ef8a29ed098777bd3e813be05","Type":"ContainerDied","Data":"14d084ee274713bc6669fe17c4f3e4d88b1c22eefda56065fa0103aa3f47986a"} Nov 04 00:57:34.799263 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.799241 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" event={"ID":"f7d4196ef8a29ed098777bd3e813be05","Type":"ContainerStarted","Data":"abdcb4619f16a6dd66cf8370e60cfcbd1618f909b34cad8d6dd00a0ea97b6691"} Nov 04 00:57:34.799297 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.799288 5786 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:34.799297 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.799294 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" event={"ID":"48bdaec858a4ad8070f76d6d06532a70","Type":"ContainerStarted","Data":"288e545b565fd5c17b551208aba6cab8cc14327e3fde88a95b27c2c029e7005a"} Nov 04 00:57:34.799351 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.799309 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" event={"ID":"48bdaec858a4ad8070f76d6d06532a70","Type":"ContainerStarted","Data":"6ea2674aeefe26451bfec13e94671d6b93ea1ddb36e50af31affbb530e658065"} Nov 04 00:57:34.858266 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.858234 5786 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:34.858412 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.858315 5786 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:34.858926 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.858898 5786 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:34.858926 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.858916 5786 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:34.859064 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:57:34.858958 5786 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:34.859064 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:57:34.858960 5786 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:34.956242 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.956205 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48bdaec858a4ad8070f76d6d06532a70-config\") pod \"kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"48bdaec858a4ad8070f76d6d06532a70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:34.956242 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.956243 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7d4196ef8a29ed098777bd3e813be05-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"f7d4196ef8a29ed098777bd3e813be05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:34.956429 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:34.956262 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7d4196ef8a29ed098777bd3e813be05-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"f7d4196ef8a29ed098777bd3e813be05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:35.057368 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.057289 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7d4196ef8a29ed098777bd3e813be05-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"f7d4196ef8a29ed098777bd3e813be05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:35.057368 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.057244 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7d4196ef8a29ed098777bd3e813be05-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"f7d4196ef8a29ed098777bd3e813be05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:35.057368 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.057365 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7d4196ef8a29ed098777bd3e813be05-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"f7d4196ef8a29ed098777bd3e813be05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:35.057574 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.057385 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48bdaec858a4ad8070f76d6d06532a70-config\") pod \"kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"48bdaec858a4ad8070f76d6d06532a70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:35.057574 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.057424 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48bdaec858a4ad8070f76d6d06532a70-config\") pod \"kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"48bdaec858a4ad8070f76d6d06532a70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:35.057574 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.057448 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7d4196ef8a29ed098777bd3e813be05-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" (UID: \"f7d4196ef8a29ed098777bd3e813be05\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:35.452071 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.452034 5786 apiserver.go:52] "Watching apiserver" Nov 04 00:57:35.552353 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.552322 5786 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 00:57:35.553030 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.553003 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-rvxjh","openshift-network-console/networking-console-plugin-86c478bb88-wkh6k","kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal","kube-system/kas-connection-checker-685895c7c9-bnf5w","openshift-network-diagnostics/network-check-target-9slf7","openshift-multus/multus-jznmq","openshift-console-operator/console-operator-7cd56b8dbc-478s5","openshift-image-registry/image-registry-559546cdcc-tvmhg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq","openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-9z6k8","openshift-image-registry/node-ca-vbxr9","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh","openshift-cluster-node-tuning-operator/tuned-dr657","openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6","openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww","openshift-dns/dns-default-zm7jt","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn","openshift-insights/insights-runtime-extractor-w6x6k","openshift-insights/insights-operator-7c4986d449-s85v2","openshift-multus/multus-additional-cni-plugins-wvng4","openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq","kube-system/global-pull-secret-syncer-pclxc","openshift-network-operator/iptables-alerter-ps7mf","openshift-ingress/router-default-767db4f785-j87sp","kube-system/kas-connection-checker-685895c7c9-gwvrx","openshift-image-registry/image-registry-5768bf545b-g67zq","openshift-multus/network-metrics-daemon-z5rxm","kube-system/kas-connection-checker-685895c7c9-knfg8","openshift-ingress-canary/ingress-canary-dwkfx","kube-system/konnectivity-agent-7smhm"] Nov 04 00:57:35.570543 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.570517 5786 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:35.570686 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.570643 5786 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:35.604087 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.604050 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.604087 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.604050 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.605736 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.605714 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 04 00:57:35.605849 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.605796 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 04 00:57:35.605849 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.605799 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 04 00:57:35.605849 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.605799 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-b7hfb\"" Nov 04 00:57:35.605849 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.605818 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.606020 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.605906 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 04 00:57:35.606691 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.606582 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.606691 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.606673 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.606899 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.606719 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.606899 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.606858 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wnwfz\"" Nov 04 00:57:35.607238 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.607220 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 04 00:57:35.607390 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.607365 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 04 00:57:35.607457 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.607338 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.607614 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.607290 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.607859 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.607840 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.607958 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.607927 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 04 00:57:35.608058 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.608021 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 04 00:57:35.608256 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.608237 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5hjbn\"" Nov 04 00:57:35.608346 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.608259 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 04 00:57:35.608346 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.608241 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.608993 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.608974 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 04 00:57:35.609088 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.609069 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.661390 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661353 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:57:35.661390 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661391 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovnkube-config\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.661587 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661412 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.661587 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661429 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/762f4cf8-0912-4663-94e4-ef0c84740817-host\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:57:35.661587 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661551 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-sys\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.661710 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661637 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psm57\" (UniqueName: \"kubernetes.io/projected/593868da-9895-4d98-9da1-4fa815c11674-kube-api-access-psm57\") pod \"kas-connection-checker-685895c7c9-bnf5w\" (UID: \"593868da-9895-4d98-9da1-4fa815c11674\") " pod="kube-system/kas-connection-checker-685895c7c9-bnf5w" Nov 04 00:57:35.661710 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661672 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/011d3a1a-a2a5-4ef3-a871-453fb22817d5-tmp-dir\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:57:35.661710 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661705 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/21952e92-3601-4969-adbd-6c6ff9b87074-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:57:35.661852 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661731 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/68909b0c-2aad-4e3f-ab79-d13d21561db5-snapshots\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:57:35.661852 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661741 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/21952e92-3601-4969-adbd-6c6ff9b87074-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:57:35.661852 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661760 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-registration-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.661852 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661784 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/762f4cf8-0912-4663-94e4-ef0c84740817-serviceca\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:57:35.661852 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661805 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/011d3a1a-a2a5-4ef3-a871-453fb22817d5-tmp-dir\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:57:35.662000 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661888 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-k8s-cni-cncf-io\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.662000 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661909 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-image-registry-private-configuration\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.662000 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661913 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/68909b0c-2aad-4e3f-ab79-d13d21561db5-snapshots\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:57:35.662000 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661945 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cnibin\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.662000 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661971 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-sys-fs\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.662000 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.661987 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-system-cni-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.662226 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662036 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkx26\" (UniqueName: \"kubernetes.io/projected/78bd204f-98ed-42b0-9c0d-97baf9aa090d-kube-api-access-fkx26\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:57:35.662226 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662054 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-certificates\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.662226 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662070 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:35.662226 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662088 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovnkube-script-lib\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.662226 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662104 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-lib-modules\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.662226 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662143 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-system-cni-dir\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.662226 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662208 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cni-binary-copy\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.662415 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662253 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-etc-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.662415 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662316 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:57:35.662415 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662393 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-hostroot\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.662504 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662444 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68909b0c-2aad-4e3f-ab79-d13d21561db5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:57:35.662504 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662461 5786 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:35.662504 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662469 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-tuned\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.662597 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662513 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/576edbd7-09bc-44d8-ba89-12af92909132-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:57:35.662597 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:57:35.662526 5786 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:35.662597 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662542 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-etc-kubernetes\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.662597 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662547 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-tuned\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.662597 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662567 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-systemd-units\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.662597 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662593 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-systemd\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.662779 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662663 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-bound-sa-token\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.662779 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662714 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-netns\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.662841 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662818 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zkfp\" (UniqueName: \"kubernetes.io/projected/3181e3c4-6f85-4302-965b-672bbf7534ba-kube-api-access-6zkfp\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:57:35.662877 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662840 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/576edbd7-09bc-44d8-ba89-12af92909132-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:57:35.662919 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662908 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:57:35.662953 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.662944 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2534417-f751-4241-b22d-3ea465dc6a19-serving-cert\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:35.663029 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663013 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-var-lib-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.663132 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663101 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-trusted-ca\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.663179 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663156 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:57:35.663213 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663189 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.663250 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663214 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4p4d\" (UniqueName: \"kubernetes.io/projected/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-kube-api-access-h4p4d\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.663250 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663242 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b9md\" (UniqueName: \"kubernetes.io/projected/21952e92-3601-4969-adbd-6c6ff9b87074-kube-api-access-2b9md\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:57:35.663366 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663347 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.663486 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663371 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-modprobe-d\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.663486 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663399 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceb1e5e7-27de-4c20-9576-52d3989384b7-cni-binary-copy\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.663486 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663403 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2534417-f751-4241-b22d-3ea465dc6a19-serving-cert\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:35.663486 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663424 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-daemon-config\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.663693 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663489 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2534417-f751-4241-b22d-3ea465dc6a19-trusted-ca\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:35.663693 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663523 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-log-socket\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.663693 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663556 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-registry-certificates\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.663693 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663638 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.663693 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663664 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:57:35.663693 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663680 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2534417-f751-4241-b22d-3ea465dc6a19-config\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:35.663693 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663696 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-node-log\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663713 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-bound-sa-token\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663729 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc65g\" (UniqueName: \"kubernetes.io/projected/2c89163b-5d76-4794-93bb-56840ed333d7-kube-api-access-jc65g\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663745 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-multus-certs\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663763 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3181e3c4-6f85-4302-965b-672bbf7534ba-iptables-alerter-script\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663789 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd0fcf4-1163-444a-95cd-be27096da34d-tmp-dir\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663827 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f84f08f4-67c2-48b5-b939-ab9269a43bfc-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663877 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2534417-f751-4241-b22d-3ea465dc6a19-trusted-ca\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663896 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d15630f9-042e-4c31-8f70-c8733c9dc31e-konnectivity-ca\") pod \"konnectivity-agent-7smhm\" (UID: \"d15630f9-042e-4c31-8f70-c8733c9dc31e\") " pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663950 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2534417-f751-4241-b22d-3ea465dc6a19-config\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663951 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd0fcf4-1163-444a-95cd-be27096da34d-tmp-dir\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663963 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d15630f9-042e-4c31-8f70-c8733c9dc31e-agent-certs\") pod \"konnectivity-agent-7smhm\" (UID: \"d15630f9-042e-4c31-8f70-c8733c9dc31e\") " pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.663990 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-kubernetes\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664007 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3181e3c4-6f85-4302-965b-672bbf7534ba-host-slash\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:57:35.664032 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664031 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-cni-bin\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664064 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664090 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbd1d155-f232-4679-a70a-ac3911aeb391-ca-trust-extracted\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664152 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664175 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbd1d155-f232-4679-a70a-ac3911aeb391-ca-trust-extracted\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664202 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac49addd-a050-4d12-bdc0-20345e446a15-ca-trust-extracted\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664230 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd0fcf4-1163-444a-95cd-be27096da34d-config-volume\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664248 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664295 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac49addd-a050-4d12-bdc0-20345e446a15-ca-trust-extracted\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664297 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-installation-pull-secrets\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664327 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-var-lib-kubelet\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664344 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-cnibin\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664360 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b64026c1-aa80-4092-829e-91e5e8164afe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664384 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovn-node-metrics-cert\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664427 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-image-registry-private-configuration\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664450 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6kt\" (UniqueName: \"kubernetes.io/projected/2248ed8a-1d14-4a43-8e03-7bf83ed903a6-kube-api-access-gq6kt\") pod \"network-check-source-6f78d59f99-cm7m6\" (UID: \"2248ed8a-1d14-4a43-8e03-7bf83ed903a6\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664466 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df3f7f74-c7a1-4f0d-8328-39f4047b6424-tmp\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664486 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-cni-bin\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664503 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-cni-multus\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.664516 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664519 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68909b0c-2aad-4e3f-ab79-d13d21561db5-service-ca-bundle\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664537 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-kubelet\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664561 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/576edbd7-09bc-44d8-ba89-12af92909132-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664562 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df3f7f74-c7a1-4f0d-8328-39f4047b6424-tmp\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664582 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664616 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-conf-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664644 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2095b82-50e2-40c5-804e-4f91d84d5e03-config\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664666 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-device-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664706 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-default-certificate\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664727 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-ovn\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664729 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/576edbd7-09bc-44d8-ba89-12af92909132-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664745 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-env-overrides\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664760 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ebe7f81-60bb-423b-a871-c2bab48db5c3-dbus\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664805 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysconfig\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664839 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2095b82-50e2-40c5-804e-4f91d84d5e03-config\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664848 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff8l8\" (UniqueName: \"kubernetes.io/projected/df3f7f74-c7a1-4f0d-8328-39f4047b6424-kube-api-access-ff8l8\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664880 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664910 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-systemd\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664936 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.664975 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-socket-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665016 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8s82\" (UniqueName: \"kubernetes.io/projected/2fa177a8-889a-4a1e-9616-7d2f81a55c8b-kube-api-access-g8s82\") pod \"volume-data-source-validator-58c9f5f46d-w9wdc\" (UID: \"2fa177a8-889a-4a1e-9616-7d2f81a55c8b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665036 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-stats-auth\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665052 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llcg6\" (UniqueName: \"kubernetes.io/projected/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-kube-api-access-llcg6\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665069 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665086 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7sjz\" (UniqueName: \"kubernetes.io/projected/ceb1e5e7-27de-4c20-9576-52d3989384b7-kube-api-access-n7sjz\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665104 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpl2n\" (UniqueName: \"kubernetes.io/projected/b2095b82-50e2-40c5-804e-4f91d84d5e03-kube-api-access-cpl2n\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665149 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljcp\" (UniqueName: \"kubernetes.io/projected/011d3a1a-a2a5-4ef3-a871-453fb22817d5-kube-api-access-jljcp\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665174 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-run-netns\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665190 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-cni-netd\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665205 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysctl-d\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665226 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qb9d\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-kube-api-access-6qb9d\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665282 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t95k\" (UniqueName: \"kubernetes.io/projected/81991817-9603-48e1-b25f-0392e58a83b5-kube-api-access-2t95k\") pod \"kas-connection-checker-685895c7c9-knfg8\" (UID: \"81991817-9603-48e1-b25f-0392e58a83b5\") " pod="kube-system/kas-connection-checker-685895c7c9-knfg8" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665318 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74c2k\" (UniqueName: \"kubernetes.io/projected/576edbd7-09bc-44d8-ba89-12af92909132-kube-api-access-74c2k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:57:35.665350 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665350 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-socket-dir-parent\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665379 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7clq\" (UniqueName: \"kubernetes.io/projected/f84f08f4-67c2-48b5-b939-ab9269a43bfc-kube-api-access-j7clq\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665385 5786 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665406 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2095b82-50e2-40c5-804e-4f91d84d5e03-serving-cert\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665446 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665466 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-run\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:57:35.665516 5786 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-94.us-west-2.compute.internal" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665520 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqt8\" (UniqueName: \"kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8\") pod \"network-check-target-9slf7\" (UID: \"7a4f60af-259e-442f-a17b-911beb60fee5\") " pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665572 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-kubelet\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665646 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68909b0c-2aad-4e3f-ab79-d13d21561db5-tmp\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665742 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665761 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb6fq\" (UniqueName: \"kubernetes.io/projected/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-kube-api-access-mb6fq\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665776 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2095b82-50e2-40c5-804e-4f91d84d5e03-serving-cert\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665778 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47n8\" (UniqueName: \"kubernetes.io/projected/762f4cf8-0912-4663-94e4-ef0c84740817-kube-api-access-n47n8\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665877 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-host\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665894 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ebe7f81-60bb-423b-a871-c2bab48db5c3-kubelet-config\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665934 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68909b0c-2aad-4e3f-ab79-d13d21561db5-tmp\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665946 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68909b0c-2aad-4e3f-ab79-d13d21561db5-serving-cert\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665973 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7hs\" (UniqueName: \"kubernetes.io/projected/b098c99f-60e8-40bd-a431-fa252a476a0b-kube-api-access-nt7hs\") pod \"kas-connection-checker-685895c7c9-gwvrx\" (UID: \"b098c99f-60e8-40bd-a431-fa252a476a0b\") " pod="kube-system/kas-connection-checker-685895c7c9-gwvrx" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.665990 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-installation-pull-secrets\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.666062 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvl2j\" (UniqueName: \"kubernetes.io/projected/68909b0c-2aad-4e3f-ab79-d13d21561db5-kube-api-access-bvl2j\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.666108 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/011d3a1a-a2a5-4ef3-a871-453fb22817d5-hosts-file\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.666151 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44f4\" (UniqueName: \"kubernetes.io/projected/3bd0fcf4-1163-444a-95cd-be27096da34d-kube-api-access-q44f4\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.666178 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgql6\" (UniqueName: \"kubernetes.io/projected/e2534417-f751-4241-b22d-3ea465dc6a19-kube-api-access-zgql6\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.666221 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.666249 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-slash\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.666274 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysctl-conf\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.666347 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-trusted-ca\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.666385 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dm7x\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-kube-api-access-4dm7x\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.666733 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.666424 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-cni-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.667912 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.667897 5786 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 00:57:35.704677 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.704572 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.704677 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.704631 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.704873 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.704859 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.704942 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.704901 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 00:57:35.704942 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.704924 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.705060 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705042 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 00:57:35.705310 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705294 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.705404 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705313 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.705404 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705356 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.705642 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705514 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.705642 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705551 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 04 00:57:35.705642 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705593 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.705838 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705647 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.705838 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705652 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.705838 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705714 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.705838 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705758 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.705838 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705778 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 00:57:35.705838 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705798 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.706069 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705850 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 00:57:35.706069 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705864 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.706069 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705926 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 00:57:35.706069 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.705930 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.706380 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706093 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.706380 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706157 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.706380 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706233 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 00:57:35.706380 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706244 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.706380 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706280 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.706380 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706292 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 00:57:35.706380 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706299 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.706699 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706488 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.706699 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706539 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.706862 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706758 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 00:57:35.706862 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706787 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 00:57:35.706862 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706788 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 00:57:35.706862 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706821 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 00:57:35.706862 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706839 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.706862 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706852 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.707321 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706869 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 04 00:57:35.707321 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706878 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.707321 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.706999 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 00:57:35.707321 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.707114 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 00:57:35.707321 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.707214 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.707558 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.707355 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 00:57:35.707785 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.707752 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 04 00:57:35.707881 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.707808 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 04 00:57:35.708172 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708139 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 00:57:35.708172 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708166 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 00:57:35.708529 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708174 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qxmlz\"" Nov 04 00:57:35.708529 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708177 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4d9rj\"" Nov 04 00:57:35.708529 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708275 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xq5nc\"" Nov 04 00:57:35.708529 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708302 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-47hrz\"" Nov 04 00:57:35.708529 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708401 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-p4bs4\"" Nov 04 00:57:35.708529 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708402 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 00:57:35.708529 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708422 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7knl\"" Nov 04 00:57:35.708529 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708448 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9rklk\"" Nov 04 00:57:35.708529 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708460 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5lbwr\"" Nov 04 00:57:35.708529 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708469 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 00:57:35.708529 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708472 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2cfvb\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708561 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708578 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fh9wk\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708600 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8chh7\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708611 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ql7fx\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708601 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708649 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708647 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-66h22\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708666 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-djbdf\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708647 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tdv96\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708678 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708674 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gczmn\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708691 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708684 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708770 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6gr6q\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708777 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708790 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w95zk\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708792 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6tf45\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708820 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-g2z88\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708904 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708944 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9lbk4\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.708983 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 04 00:57:35.709059 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.709037 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2w2b9\"" Nov 04 00:57:35.709628 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.709077 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 00:57:35.712413 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.712392 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovnkube-config\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.712499 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.712483 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cni-binary-copy\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.712553 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.712507 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-metrics-certs\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:35.712553 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.712483 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovnkube-script-lib\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.712658 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.712547 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-image-registry-private-configuration\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.712658 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.712588 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-certificates\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.712658 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.712631 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/21952e92-3601-4969-adbd-6c6ff9b87074-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:57:35.712774 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.712753 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ebe7f81-60bb-423b-a871-c2bab48db5c3-original-pull-secret\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:57:35.712842 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.712827 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/762f4cf8-0912-4663-94e4-ef0c84740817-serviceca\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:57:35.713602 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.713587 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceb1e5e7-27de-4c20-9576-52d3989384b7-cni-binary-copy\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.713687 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.713633 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-daemon-config\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.713859 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.713841 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.714241 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714220 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-registry-certificates\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.714408 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714391 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-service-ca-bundle\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:35.714522 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714406 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3181e3c4-6f85-4302-965b-672bbf7534ba-iptables-alerter-script\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:57:35.714522 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714429 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d15630f9-042e-4c31-8f70-c8733c9dc31e-konnectivity-ca\") pod \"konnectivity-agent-7smhm\" (UID: \"d15630f9-042e-4c31-8f70-c8733c9dc31e\") " pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:57:35.714522 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714477 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd0fcf4-1163-444a-95cd-be27096da34d-config-volume\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:35.714676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714550 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b64026c1-aa80-4092-829e-91e5e8164afe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:57:35.714676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714548 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-installation-pull-secrets\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.714676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714580 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d15630f9-042e-4c31-8f70-c8733c9dc31e-agent-certs\") pod \"konnectivity-agent-7smhm\" (UID: \"d15630f9-042e-4c31-8f70-c8733c9dc31e\") " pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:57:35.714676 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714635 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-image-registry-private-configuration\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.714789 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714674 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-ovn-node-metrics-cert\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.714822 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714803 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.714933 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714918 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-env-overrides\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.714966 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714948 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68909b0c-2aad-4e3f-ab79-d13d21561db5-service-ca-bundle\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:57:35.714966 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.714956 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-default-certificate\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:35.715968 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.715948 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b64026c1-aa80-4092-829e-91e5e8164afe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wkh6k\" (UID: \"b64026c1-aa80-4092-829e-91e5e8164afe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wkh6k" Nov 04 00:57:35.716031 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.715968 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.716031 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.716026 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3bd0fcf4-1163-444a-95cd-be27096da34d-metrics-tls\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:35.716190 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.716172 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.716241 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.716194 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-stats-auth\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:35.716241 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.716198 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-installation-pull-secrets\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.716349 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.716335 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68909b0c-2aad-4e3f-ab79-d13d21561db5-serving-cert\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:57:35.716472 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.716460 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bd204f-98ed-42b0-9c0d-97baf9aa090d-metrics-certs\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:57:35.754181 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.754047 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 00:57:35.754815 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.754780 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zkfp\" (UniqueName: \"kubernetes.io/projected/3181e3c4-6f85-4302-965b-672bbf7534ba-kube-api-access-6zkfp\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:57:35.754815 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.754803 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkx26\" (UniqueName: \"kubernetes.io/projected/78bd204f-98ed-42b0-9c0d-97baf9aa090d-kube-api-access-fkx26\") pod \"network-metrics-daemon-z5rxm\" (UID: \"78bd204f-98ed-42b0-9c0d-97baf9aa090d\") " pod="openshift-multus/network-metrics-daemon-z5rxm" Nov 04 00:57:35.755072 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.754791 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc65g\" (UniqueName: \"kubernetes.io/projected/2c89163b-5d76-4794-93bb-56840ed333d7-kube-api-access-jc65g\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.755072 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.754929 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-bound-sa-token\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.755388 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.755365 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-bound-sa-token\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.755492 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.755470 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74c2k\" (UniqueName: \"kubernetes.io/projected/576edbd7-09bc-44d8-ba89-12af92909132-kube-api-access-74c2k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh\" (UID: \"576edbd7-09bc-44d8-ba89-12af92909132\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" Nov 04 00:57:35.755596 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.755549 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljcp\" (UniqueName: \"kubernetes.io/projected/011d3a1a-a2a5-4ef3-a871-453fb22817d5-kube-api-access-jljcp\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:57:35.755637 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.755592 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4p4d\" (UniqueName: \"kubernetes.io/projected/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-kube-api-access-h4p4d\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.755846 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.755823 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dm7x\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-kube-api-access-4dm7x\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.756162 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.756144 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44f4\" (UniqueName: \"kubernetes.io/projected/3bd0fcf4-1163-444a-95cd-be27096da34d-kube-api-access-q44f4\") pod \"dns-default-zm7jt\" (UID: \"3bd0fcf4-1163-444a-95cd-be27096da34d\") " pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:35.756382 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.756360 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 00:57:35.756690 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.756670 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llcg6\" (UniqueName: \"kubernetes.io/projected/28aa6fa1-33f8-450e-a1f1-d382e7cf6a47-kube-api-access-llcg6\") pod \"router-default-767db4f785-j87sp\" (UID: \"28aa6fa1-33f8-450e-a1f1-d382e7cf6a47\") " pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:35.756742 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.756684 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psm57\" (UniqueName: \"kubernetes.io/projected/593868da-9895-4d98-9da1-4fa815c11674-kube-api-access-psm57\") pod \"kas-connection-checker-685895c7c9-bnf5w\" (UID: \"593868da-9895-4d98-9da1-4fa815c11674\") " pod="kube-system/kas-connection-checker-685895c7c9-bnf5w" Nov 04 00:57:35.756742 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.756682 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb6fq\" (UniqueName: \"kubernetes.io/projected/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-kube-api-access-mb6fq\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.756942 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.756918 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpl2n\" (UniqueName: \"kubernetes.io/projected/b2095b82-50e2-40c5-804e-4f91d84d5e03-kube-api-access-cpl2n\") pod \"service-ca-operator-544d749c8c-5lcww\" (UID: \"b2095b82-50e2-40c5-804e-4f91d84d5e03\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" Nov 04 00:57:35.756992 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.756920 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7clq\" (UniqueName: \"kubernetes.io/projected/f84f08f4-67c2-48b5-b939-ab9269a43bfc-kube-api-access-j7clq\") pod \"cluster-samples-operator-6ddb6c8b94-n4pvn\" (UID: \"f84f08f4-67c2-48b5-b939-ab9269a43bfc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n4pvn" Nov 04 00:57:35.757532 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.757483 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b9md\" (UniqueName: \"kubernetes.io/projected/21952e92-3601-4969-adbd-6c6ff9b87074-kube-api-access-2b9md\") pod \"cluster-monitoring-operator-884576c57-4bql4\" (UID: \"21952e92-3601-4969-adbd-6c6ff9b87074\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bql4" Nov 04 00:57:35.757632 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.757541 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7hs\" (UniqueName: \"kubernetes.io/projected/b098c99f-60e8-40bd-a431-fa252a476a0b-kube-api-access-nt7hs\") pod \"kas-connection-checker-685895c7c9-gwvrx\" (UID: \"b098c99f-60e8-40bd-a431-fa252a476a0b\") " pod="kube-system/kas-connection-checker-685895c7c9-gwvrx" Nov 04 00:57:35.758256 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.757877 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7sjz\" (UniqueName: \"kubernetes.io/projected/ceb1e5e7-27de-4c20-9576-52d3989384b7-kube-api-access-n7sjz\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.758256 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.757970 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff8l8\" (UniqueName: \"kubernetes.io/projected/df3f7f74-c7a1-4f0d-8328-39f4047b6424-kube-api-access-ff8l8\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.758256 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.758027 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-trusted-ca\") pod \"image-registry-559546cdcc-tvmhg\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:57:35.758256 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.758041 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvl2j\" (UniqueName: \"kubernetes.io/projected/68909b0c-2aad-4e3f-ab79-d13d21561db5-kube-api-access-bvl2j\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:57:35.758256 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.758173 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qb9d\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-kube-api-access-6qb9d\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.758256 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.758197 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47n8\" (UniqueName: \"kubernetes.io/projected/762f4cf8-0912-4663-94e4-ef0c84740817-kube-api-access-n47n8\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:57:35.758256 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.758212 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scqt8\" (UniqueName: \"kubernetes.io/projected/7a4f60af-259e-442f-a17b-911beb60fee5-kube-api-access-scqt8\") pod \"network-check-target-9slf7\" (UID: \"7a4f60af-259e-442f-a17b-911beb60fee5\") " pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:57:35.758602 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.758299 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8s82\" (UniqueName: \"kubernetes.io/projected/2fa177a8-889a-4a1e-9616-7d2f81a55c8b-kube-api-access-g8s82\") pod \"volume-data-source-validator-58c9f5f46d-w9wdc\" (UID: \"2fa177a8-889a-4a1e-9616-7d2f81a55c8b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w9wdc" Nov 04 00:57:35.758602 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.758449 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6kt\" (UniqueName: \"kubernetes.io/projected/2248ed8a-1d14-4a43-8e03-7bf83ed903a6-kube-api-access-gq6kt\") pod \"network-check-source-6f78d59f99-cm7m6\" (UID: \"2248ed8a-1d14-4a43-8e03-7bf83ed903a6\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm7m6" Nov 04 00:57:35.758602 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.758536 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t95k\" (UniqueName: \"kubernetes.io/projected/81991817-9603-48e1-b25f-0392e58a83b5-kube-api-access-2t95k\") pod \"kas-connection-checker-685895c7c9-knfg8\" (UID: \"81991817-9603-48e1-b25f-0392e58a83b5\") " pod="kube-system/kas-connection-checker-685895c7c9-knfg8" Nov 04 00:57:35.758701 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.758630 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgql6\" (UniqueName: \"kubernetes.io/projected/e2534417-f751-4241-b22d-3ea465dc6a19-kube-api-access-zgql6\") pod \"console-operator-7cd56b8dbc-478s5\" (UID: \"e2534417-f751-4241-b22d-3ea465dc6a19\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:35.763415 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.763386 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68909b0c-2aad-4e3f-ab79-d13d21561db5-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-s85v2\" (UID: \"68909b0c-2aad-4e3f-ab79-d13d21561db5\") " pod="openshift-insights/insights-operator-7c4986d449-s85v2" Nov 04 00:57:35.767558 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767528 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-trusted-ca\") pod \"image-registry-5768bf545b-g67zq\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:35.767558 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767541 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-socket-dir-parent\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.767667 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767591 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-socket-dir-parent\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.767667 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767591 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.767667 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767629 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-run\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.767667 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767641 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.767667 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767649 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-kubelet\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.767667 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767667 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-kubelet\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767686 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-host\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767704 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ebe7f81-60bb-423b-a871-c2bab48db5c3-kubelet-config\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767705 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-run\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767721 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/011d3a1a-a2a5-4ef3-a871-453fb22817d5-hosts-file\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767738 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8cn\" (UniqueName: \"kubernetes.io/projected/f8fe96da-a52f-4f6c-ae1d-229955f4d9ec-kube-api-access-jh8cn\") pod \"migrator-66df5d496b-2whjq\" (UID: \"f8fe96da-a52f-4f6c-ae1d-229955f4d9ec\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767739 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-host\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767752 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-slash\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767752 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ebe7f81-60bb-423b-a871-c2bab48db5c3-kubelet-config\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767776 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-slash\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767781 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysctl-conf\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767812 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-cni-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767819 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/011d3a1a-a2a5-4ef3-a871-453fb22817d5-hosts-file\") pod \"node-resolver-rvxjh\" (UID: \"011d3a1a-a2a5-4ef3-a871-453fb22817d5\") " pod="openshift-dns/node-resolver-rvxjh" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767835 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767867 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/762f4cf8-0912-4663-94e4-ef0c84740817-host\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767879 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767888 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-sys\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767891 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysctl-conf\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767912 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-registration-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767911 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-cni-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767921 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/762f4cf8-0912-4663-94e4-ef0c84740817-host\") pod \"node-ca-vbxr9\" (UID: \"762f4cf8-0912-4663-94e4-ef0c84740817\") " pod="openshift-image-registry/node-ca-vbxr9" Nov 04 00:57:35.767929 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767940 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-sys\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767940 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f5b81fe1-bb35-4200-9b69-c49ed020ca64-crio-socket\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767970 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-registration-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.767981 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-k8s-cni-cncf-io\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768013 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-k8s-cni-cncf-io\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768088 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cnibin\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768137 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-cnibin\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768148 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-sys-fs\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768181 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768198 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-sys-fs\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768214 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-system-cni-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768237 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-lib-modules\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768252 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-system-cni-dir\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768267 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-etc-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768284 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-hostroot\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768308 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-etc-kubernetes\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768316 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-system-cni-dir\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768348 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-system-cni-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768358 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768364 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-lib-modules\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768370 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-systemd-units\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768352 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-etc-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768364 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-hostroot\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768354 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-etc-kubernetes\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768399 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-systemd\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768405 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-systemd-units\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768416 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-netns\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768438 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-var-lib-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768465 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e-cert\") pod \"ingress-canary-dwkfx\" (UID: \"40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e\") " pod="openshift-ingress-canary/ingress-canary-dwkfx" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768471 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-systemd\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768480 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-var-lib-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768467 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-netns\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768488 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768511 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768525 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-modprobe-d\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768541 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-log-socket\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768547 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ad9dfd7-097e-4491-a3b3-ba3c89a52992-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wvng4\" (UID: \"2ad9dfd7-097e-4491-a3b3-ba3c89a52992\") " pod="openshift-multus/multus-additional-cni-plugins-wvng4" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768559 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-node-log\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768584 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768600 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-node-log\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768600 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-log-socket\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768618 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-multus-certs\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768647 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-kubernetes\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768649 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-modprobe-d\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768671 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3181e3c4-6f85-4302-965b-672bbf7534ba-host-slash\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768680 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e-cert\") pod \"ingress-canary-dwkfx\" (UID: \"40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e\") " pod="openshift-ingress-canary/ingress-canary-dwkfx" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768685 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-kubernetes\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768695 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-cni-bin\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768701 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3181e3c4-6f85-4302-965b-672bbf7534ba-host-slash\") pod \"iptables-alerter-ps7mf\" (UID: \"3181e3c4-6f85-4302-965b-672bbf7534ba\") " pod="openshift-network-operator/iptables-alerter-ps7mf" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768649 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-run-multus-certs\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768721 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.768726 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768745 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768723 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-cni-bin\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768767 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-var-lib-kubelet\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768787 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-openvswitch\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768804 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-cnibin\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768816 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-var-lib-kubelet\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768835 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768859 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768862 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-cni-bin\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768872 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-cnibin\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768900 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-cni-bin\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768927 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-cni-multus\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768955 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5b81fe1-bb35-4200-9b69-c49ed020ca64-data-volume\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.768979 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-kubelet\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769006 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-host-var-lib-cni-multus\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769004 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-conf-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769034 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-device-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769041 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-kubelet\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769049 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-ovn\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769042 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceb1e5e7-27de-4c20-9576-52d3989384b7-multus-conf-dir\") pod \"multus-jznmq\" (UID: \"ceb1e5e7-27de-4c20-9576-52d3989384b7\") " pod="openshift-multus/multus-jznmq" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769064 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ebe7f81-60bb-423b-a871-c2bab48db5c3-dbus\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769069 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5b81fe1-bb35-4200-9b69-c49ed020ca64-data-volume\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769078 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-ovn\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769062 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5b81fe1-bb35-4200-9b69-c49ed020ca64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769080 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wtnl\" (UniqueName: \"kubernetes.io/projected/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e-kube-api-access-5wtnl\") pod \"ingress-canary-dwkfx\" (UID: \"40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e\") " pod="openshift-ingress-canary/ingress-canary-dwkfx" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769109 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-device-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769109 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ebe7f81-60bb-423b-a871-c2bab48db5c3-dbus\") pod \"global-pull-secret-syncer-pclxc\" (UID: \"0ebe7f81-60bb-423b-a871-c2bab48db5c3\") " pod="kube-system/global-pull-secret-syncer-pclxc" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769133 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysconfig\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769161 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysconfig\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769166 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-systemd\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769182 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-socket-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769186 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-run-systemd\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769198 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-run-netns\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769212 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-cni-netd\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769225 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysctl-d\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769243 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thr9\" (UniqueName: \"kubernetes.io/projected/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-api-access-7thr9\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769243 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-run-netns\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769226 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c89163b-5d76-4794-93bb-56840ed333d7-socket-dir\") pod \"aws-ebs-csi-driver-node-r7zlq\" (UID: \"2c89163b-5d76-4794-93bb-56840ed333d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r7zlq" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769284 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d69ef80-a3b8-4c02-a636-6b30e21e5b2a-host-cni-netd\") pod \"ovnkube-node-9z6k8\" (UID: \"6d69ef80-a3b8-4c02-a636-6b30e21e5b2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:35.770306 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.769321 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df3f7f74-c7a1-4f0d-8328-39f4047b6424-etc-sysctl-d\") pod \"tuned-dr657\" (UID: \"df3f7f74-c7a1-4f0d-8328-39f4047b6424\") " pod="openshift-cluster-node-tuning-operator/tuned-dr657" Nov 04 00:57:35.820890 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.820858 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8cn\" (UniqueName: \"kubernetes.io/projected/f8fe96da-a52f-4f6c-ae1d-229955f4d9ec-kube-api-access-jh8cn\") pod \"migrator-66df5d496b-2whjq\" (UID: \"f8fe96da-a52f-4f6c-ae1d-229955f4d9ec\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2whjq" Nov 04 00:57:35.821565 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.821544 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wtnl\" (UniqueName: \"kubernetes.io/projected/40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e-kube-api-access-5wtnl\") pod \"ingress-canary-dwkfx\" (UID: \"40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e\") " pod="openshift-ingress-canary/ingress-canary-dwkfx" Nov 04 00:57:35.822383 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.822359 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thr9\" (UniqueName: \"kubernetes.io/projected/f5b81fe1-bb35-4200-9b69-c49ed020ca64-kube-api-access-7thr9\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:35.855266 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.855236 5786 scope.go:122] "RemoveContainer" containerID="950227cc5d7e90c9d8958582b2659aeccb3e10e1eb241adbb3125cc62fba7272" Nov 04 00:57:35.860581 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.860561 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:35.869947 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.869929 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f5b81fe1-bb35-4200-9b69-c49ed020ca64-crio-socket\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:35.870033 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.870021 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f5b81fe1-bb35-4200-9b69-c49ed020ca64-crio-socket\") pod \"insights-runtime-extractor-w6x6k\" (UID: \"f5b81fe1-bb35-4200-9b69-c49ed020ca64\") " pod="openshift-insights/insights-runtime-extractor-w6x6k" Nov 04 00:57:35.886227 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:35.886202 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:36.577267 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:36.577183 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 00:57:36.577996 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:36.577323 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" event={"ID":"e2534417-f751-4241-b22d-3ea465dc6a19","Type":"ContainerStarted","Data":"222362c88e5a81a541b5af99b33050666c14b4caea54ef11584dba32d0355004"} Nov 04 00:57:36.578948 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:36.578924 5786 generic.go:356] "Generic (PLEG): container finished" podID="68909b0c-2aad-4e3f-ab79-d13d21561db5" containerID="a1e9fb5fadf90fc7141b0da5209f22e7a8b37a179098e0390d01b75890336dda" exitCode=0 Nov 04 00:57:36.579060 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:36.579009 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-s85v2" event={"ID":"68909b0c-2aad-4e3f-ab79-d13d21561db5","Type":"ContainerDied","Data":"a1e9fb5fadf90fc7141b0da5209f22e7a8b37a179098e0390d01b75890336dda"} Nov 04 00:57:36.579701 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:36.579683 5786 scope.go:122] "RemoveContainer" containerID="a1e9fb5fadf90fc7141b0da5209f22e7a8b37a179098e0390d01b75890336dda" Nov 04 00:57:36.862492 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:36.862458 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:37.246128 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:37.246034 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:37.251047 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:37.251022 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-478s5" Nov 04 00:57:37.411012 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:37.410982 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9slf7" Nov 04 00:57:37.585348 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:37.585264 5786 generic.go:356] "Generic (PLEG): container finished" podID="b2095b82-50e2-40c5-804e-4f91d84d5e03" containerID="c389b890606a785712a9bb6372663c8315f4248618d7e87669006235b10ecae1" exitCode=0 Nov 04 00:57:37.585348 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:37.585336 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" event={"ID":"b2095b82-50e2-40c5-804e-4f91d84d5e03","Type":"ContainerDied","Data":"c389b890606a785712a9bb6372663c8315f4248618d7e87669006235b10ecae1"} Nov 04 00:57:37.585860 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:37.585695 5786 scope.go:122] "RemoveContainer" containerID="c389b890606a785712a9bb6372663c8315f4248618d7e87669006235b10ecae1" Nov 04 00:57:37.587527 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:37.587500 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-s85v2" event={"ID":"68909b0c-2aad-4e3f-ab79-d13d21561db5","Type":"ContainerStarted","Data":"d45da7bc4d38f028648cfcd4ca691ddd5c9ebadb9f1fd735c4933c66301202c6"} Nov 04 00:57:37.846322 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:37.846296 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:38.593022 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:38.592985 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5lcww" event={"ID":"b2095b82-50e2-40c5-804e-4f91d84d5e03","Type":"ContainerStarted","Data":"f7f9d76ea3a885b8a90276df8d9b4dcd18bcf1f109ef4c48a33104cf134d2033"} Nov 04 00:57:40.601706 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:40.601669 5786 generic.go:356] "Generic (PLEG): container finished" podID="576edbd7-09bc-44d8-ba89-12af92909132" containerID="6a8d058c0163889b8109e4ebf30494274cedbdb723f0731e08935371275e9daa" exitCode=0 Nov 04 00:57:40.602201 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:40.601729 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" event={"ID":"576edbd7-09bc-44d8-ba89-12af92909132","Type":"ContainerDied","Data":"6a8d058c0163889b8109e4ebf30494274cedbdb723f0731e08935371275e9daa"} Nov 04 00:57:40.602201 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:40.602143 5786 scope.go:122] "RemoveContainer" containerID="6a8d058c0163889b8109e4ebf30494274cedbdb723f0731e08935371275e9daa" Nov 04 00:57:40.986656 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:40.986571 5786 patch_prober.go:29] interesting pod/image-registry-5768bf545b-g67zq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 04 00:57:40.986656 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:40.986628 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" podUID="cbd1d155-f232-4679-a70a-ac3911aeb391" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:57:41.119103 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:41.119066 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:41.119287 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:41.119213 5786 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:57:41.120395 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:41.120373 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-767db4f785-j87sp" Nov 04 00:57:41.239320 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:41.239242 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:41.239503 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:41.239486 5786 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:57:41.607859 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:41.607829 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rbjmh" event={"ID":"576edbd7-09bc-44d8-ba89-12af92909132","Type":"ContainerStarted","Data":"ba89f259701f7b99b9ec6cde9e6fc03d12c19992a064de8b7d98828bf42e68c1"} Nov 04 00:57:41.859290 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:41.859201 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:57:41.862948 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:41.862923 5786 patch_prober.go:29] interesting pod/image-registry-5768bf545b-g67zq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 04 00:57:41.863050 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:41.862966 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" podUID="cbd1d155-f232-4679-a70a-ac3911aeb391" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:57:41.866467 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:41.866446 5786 patch_prober.go:29] interesting pod/image-registry-5768bf545b-g67zq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 04 00:57:41.866545 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:41.866477 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" podUID="cbd1d155-f232-4679-a70a-ac3911aeb391" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:57:41.960831 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:41.960794 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z6k8" Nov 04 00:57:42.285045 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.284937 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zm7jt_3bd0fcf4-1163-444a-95cd-be27096da34d/dns/0.log" Nov 04 00:57:42.392685 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.392655 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zm7jt_3bd0fcf4-1163-444a-95cd-be27096da34d/kube-rbac-proxy/0.log" Nov 04 00:57:42.531135 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.531091 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rxt2c"] Nov 04 00:57:42.533571 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.533553 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.582040 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.582008 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zm7jt" Nov 04 00:57:42.618944 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.618905 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-accelerators-collector-config\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.619349 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.618955 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ec4a7cb-76bf-4c22-8141-04417bc3a515-root\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.619349 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.619193 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-tls\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.619349 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.619245 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ec4a7cb-76bf-4c22-8141-04417bc3a515-sys\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.619349 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.619275 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4a7cb-76bf-4c22-8141-04417bc3a515-metrics-client-ca\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.619349 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.619324 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qd5z\" (UniqueName: \"kubernetes.io/projected/8ec4a7cb-76bf-4c22-8141-04417bc3a515-kube-api-access-2qd5z\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.619589 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.619365 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.619589 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.619476 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-wtmp\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.619589 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.619537 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-textfile\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.641496 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.641466 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 00:57:42.641496 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.641484 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tjpbs\"" Nov 04 00:57:42.641704 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.641629 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 00:57:42.641758 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.641723 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 00:57:42.641800 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.641760 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 00:57:42.719920 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.719882 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ec4a7cb-76bf-4c22-8141-04417bc3a515-root\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720083 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.719933 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-tls\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720083 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.719958 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ec4a7cb-76bf-4c22-8141-04417bc3a515-sys\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720083 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.719984 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4a7cb-76bf-4c22-8141-04417bc3a515-metrics-client-ca\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720083 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.719997 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ec4a7cb-76bf-4c22-8141-04417bc3a515-root\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720083 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.720000 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qd5z\" (UniqueName: \"kubernetes.io/projected/8ec4a7cb-76bf-4c22-8141-04417bc3a515-kube-api-access-2qd5z\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720083 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.720065 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ec4a7cb-76bf-4c22-8141-04417bc3a515-sys\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720083 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.720077 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720386 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.720139 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-wtmp\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720386 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.720191 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-textfile\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720386 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.720229 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-accelerators-collector-config\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720386 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.720348 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-wtmp\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720514 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.720475 5786 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 00:57:42.720514 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.720500 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-textfile\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720699 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.720671 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ec4a7cb-76bf-4c22-8141-04417bc3a515-metrics-client-ca\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.720789 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.720749 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-accelerators-collector-config\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.722559 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.722528 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.722723 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.722706 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ec4a7cb-76bf-4c22-8141-04417bc3a515-node-exporter-tls\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.775852 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.775819 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qd5z\" (UniqueName: \"kubernetes.io/projected/8ec4a7cb-76bf-4c22-8141-04417bc3a515-kube-api-access-2qd5z\") pod \"node-exporter-rxt2c\" (UID: \"8ec4a7cb-76bf-4c22-8141-04417bc3a515\") " pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.846273 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.846245 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rxt2c" Nov 04 00:57:42.857129 ip-10-0-136-94 kubenswrapper[5786]: W1104 00:57:42.857085 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec4a7cb_76bf_4c22_8141_04417bc3a515.slice/crio-40a62c32105ae421b021dad9e04924ee1fa841ca783c80bee54375419d2b5642 WatchSource:0}: Error finding container 40a62c32105ae421b021dad9e04924ee1fa841ca783c80bee54375419d2b5642: Status 404 returned error can't find the container with id 40a62c32105ae421b021dad9e04924ee1fa841ca783c80bee54375419d2b5642 Nov 04 00:57:42.859267 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.859235 5786 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:57:42.891351 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:42.891328 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rvxjh_011d3a1a-a2a5-4ef3-a871-453fb22817d5/dns-node-resolver/0.log" Nov 04 00:57:43.092811 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:43.092778 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-767db4f785-j87sp_28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/router/0.log" Nov 04 00:57:43.291764 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:43.291694 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dwkfx_40eb6c9a-7483-482b-aaa8-1cc2ca0bb93e/serve-healthcheck-canary/0.log" Nov 04 00:57:43.384575 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:43.384542 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7smhm" Nov 04 00:57:43.618995 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:43.618962 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rxt2c" event={"ID":"8ec4a7cb-76bf-4c22-8141-04417bc3a515","Type":"ContainerStarted","Data":"40a62c32105ae421b021dad9e04924ee1fa841ca783c80bee54375419d2b5642"} Nov 04 00:57:44.624587 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:44.624549 5786 generic.go:356] "Generic (PLEG): container finished" podID="8ec4a7cb-76bf-4c22-8141-04417bc3a515" containerID="00badfbcb417c69cd92a3eb9cd3ea48c25ba41a674d799cf912fa985eee95f4a" exitCode=0 Nov 04 00:57:44.624998 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:44.624627 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rxt2c" event={"ID":"8ec4a7cb-76bf-4c22-8141-04417bc3a515","Type":"ContainerDied","Data":"00badfbcb417c69cd92a3eb9cd3ea48c25ba41a674d799cf912fa985eee95f4a"} Nov 04 00:57:45.086982 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:57:45.086896 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 00:57:45.630790 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:45.630753 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rxt2c" event={"ID":"8ec4a7cb-76bf-4c22-8141-04417bc3a515","Type":"ContainerStarted","Data":"b3a7b2771434e256d06fc41ca23e47db88deef1392117ee0d6b689b5c2075e51"} Nov 04 00:57:45.631186 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:45.630807 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rxt2c" event={"ID":"8ec4a7cb-76bf-4c22-8141-04417bc3a515","Type":"ContainerStarted","Data":"cb5b5cdf594cea19a482fc634d4e537c78ae183451375c4654e494adccc31292"} Nov 04 00:57:45.742458 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:45.742387 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rxt2c" podStartSLOduration=2.127699 podStartE2EDuration="3.742374146s" totalImagesPullingTime="1.542230873s" totalInitContainerRuntime="72.444273ms" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:45.741829629 +0000 UTC m=+11.358175394" watchObservedRunningTime="2026-11-04 00:57:45.742374146 +0000 UTC m=+11.358719909" Nov 04 00:57:47.638204 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:47.638168 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5768bf545b-g67zq"] Nov 04 00:57:47.642418 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:57:47.642387 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:58:00.578536 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.578456 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" podUID="ac49addd-a050-4d12-bdc0-20345e446a15" containerName="registry" containerID="cri-o://4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62" gracePeriod=30 Nov 04 00:58:00.910074 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.910044 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:58:00.983829 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.983792 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") pod \"ac49addd-a050-4d12-bdc0-20345e446a15\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " Nov 04 00:58:00.984002 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.983846 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-kube-api-access-4dm7x\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-kube-api-access-4dm7x\") pod \"ac49addd-a050-4d12-bdc0-20345e446a15\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " Nov 04 00:58:00.984002 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.983869 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-installation-pull-secrets\") pod \"ac49addd-a050-4d12-bdc0-20345e446a15\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " Nov 04 00:58:00.984002 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.983895 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-trusted-ca\") pod \"ac49addd-a050-4d12-bdc0-20345e446a15\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " Nov 04 00:58:00.984002 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.983922 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-image-registry-private-configuration\") pod \"ac49addd-a050-4d12-bdc0-20345e446a15\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " Nov 04 00:58:00.984002 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.983964 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-registry-certificates\") pod \"ac49addd-a050-4d12-bdc0-20345e446a15\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " Nov 04 00:58:00.984279 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.984017 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-bound-sa-token\") pod \"ac49addd-a050-4d12-bdc0-20345e446a15\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " Nov 04 00:58:00.984279 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.984100 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac49addd-a050-4d12-bdc0-20345e446a15-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac49addd-a050-4d12-bdc0-20345e446a15-ca-trust-extracted\") pod \"ac49addd-a050-4d12-bdc0-20345e446a15\" (UID: \"ac49addd-a050-4d12-bdc0-20345e446a15\") " Nov 04 00:58:00.984543 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.984426 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-trusted-ca" pod "ac49addd-a050-4d12-bdc0-20345e446a15" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:00.984543 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.984483 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-registry-certificates" pod "ac49addd-a050-4d12-bdc0-20345e446a15" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:00.986499 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.986384 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls" pod "ac49addd-a050-4d12-bdc0-20345e446a15" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:00.986677 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.986536 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-image-registry-private-configuration" pod "ac49addd-a050-4d12-bdc0-20345e446a15" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:00.986677 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.986612 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-installation-pull-secrets" pod "ac49addd-a050-4d12-bdc0-20345e446a15" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:00.986793 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.986705 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-kube-api-access-4dm7x" pod "ac49addd-a050-4d12-bdc0-20345e446a15" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15"). InnerVolumeSpecName "kube-api-access-4dm7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:00.986793 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.986767 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-bound-sa-token" pod "ac49addd-a050-4d12-bdc0-20345e446a15" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:00.990914 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:00.990887 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac49addd-a050-4d12-bdc0-20345e446a15-ca-trust-extracted" pod "ac49addd-a050-4d12-bdc0-20345e446a15" (UID: "ac49addd-a050-4d12-bdc0-20345e446a15"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:58:01.085557 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.085520 5786 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-registry-certificates\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:01.085557 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.085553 5786 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-bound-sa-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:01.085557 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.085562 5786 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac49addd-a050-4d12-bdc0-20345e446a15-ca-trust-extracted\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:01.085775 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.085571 5786 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-registry-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:01.085775 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.085580 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dm7x\" (UniqueName: \"kubernetes.io/projected/ac49addd-a050-4d12-bdc0-20345e446a15-kube-api-access-4dm7x\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:01.085775 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.085589 5786 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-installation-pull-secrets\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:01.085775 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.085598 5786 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac49addd-a050-4d12-bdc0-20345e446a15-trusted-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:01.085775 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.085607 5786 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac49addd-a050-4d12-bdc0-20345e446a15-image-registry-private-configuration\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:01.688048 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.688017 5786 generic.go:356] "Generic (PLEG): container finished" podID="ac49addd-a050-4d12-bdc0-20345e446a15" containerID="4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62" exitCode=0 Nov 04 00:58:01.688484 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.688081 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" Nov 04 00:58:01.688484 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.688107 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" event={"ID":"ac49addd-a050-4d12-bdc0-20345e446a15","Type":"ContainerDied","Data":"4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62"} Nov 04 00:58:01.688484 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.688186 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-559546cdcc-tvmhg" event={"ID":"ac49addd-a050-4d12-bdc0-20345e446a15","Type":"ContainerDied","Data":"674fcbdbd6d5767b57e4719770ebe1b2a124ae6ccc3463fefe335beaa79a91d4"} Nov 04 00:58:01.688484 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.688207 5786 scope.go:122] "RemoveContainer" containerID="4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62" Nov 04 00:58:01.700798 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.700780 5786 scope.go:122] "RemoveContainer" containerID="4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62" Nov 04 00:58:01.701129 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:58:01.701099 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62\": container with ID starting with 4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62 not found: ID does not exist" containerID="4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62" Nov 04 00:58:01.701238 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.701204 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62"} err="failed to get container status \"4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62\": rpc error: code = NotFound desc = could not find container \"4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62\": container with ID starting with 4e4be724605b59bfc010b2e8a6b13eea254b5632568ea14b34b9fbf27a0e1d62 not found: ID does not exist" Nov 04 00:58:01.892767 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.892732 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-559546cdcc-tvmhg"] Nov 04 00:58:01.940439 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:01.940364 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-559546cdcc-tvmhg"] Nov 04 00:58:02.502452 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:02.502422 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac49addd-a050-4d12-bdc0-20345e446a15" path="/var/lib/kubelet/pods/ac49addd-a050-4d12-bdc0-20345e446a15/volumes" Nov 04 00:58:12.657581 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:12.657516 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" podUID="cbd1d155-f232-4679-a70a-ac3911aeb391" containerName="registry" containerID="cri-o://da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35" gracePeriod=30 Nov 04 00:58:12.910915 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:12.910863 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:58:13.086054 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.086019 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") pod \"cbd1d155-f232-4679-a70a-ac3911aeb391\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " Nov 04 00:58:13.086227 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.086088 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-kube-api-access-6qb9d\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-kube-api-access-6qb9d\") pod \"cbd1d155-f232-4679-a70a-ac3911aeb391\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " Nov 04 00:58:13.086227 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.086143 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-bound-sa-token\") pod \"cbd1d155-f232-4679-a70a-ac3911aeb391\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " Nov 04 00:58:13.086319 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.086233 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-trusted-ca\") pod \"cbd1d155-f232-4679-a70a-ac3911aeb391\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " Nov 04 00:58:13.086319 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.086271 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-installation-pull-secrets\") pod \"cbd1d155-f232-4679-a70a-ac3911aeb391\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " Nov 04 00:58:13.086319 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.086305 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-image-registry-private-configuration\") pod \"cbd1d155-f232-4679-a70a-ac3911aeb391\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " Nov 04 00:58:13.086471 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.086339 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd1d155-f232-4679-a70a-ac3911aeb391-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbd1d155-f232-4679-a70a-ac3911aeb391-ca-trust-extracted\") pod \"cbd1d155-f232-4679-a70a-ac3911aeb391\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " Nov 04 00:58:13.086471 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.086376 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-certificates\") pod \"cbd1d155-f232-4679-a70a-ac3911aeb391\" (UID: \"cbd1d155-f232-4679-a70a-ac3911aeb391\") " Nov 04 00:58:13.086875 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.086699 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-trusted-ca" pod "cbd1d155-f232-4679-a70a-ac3911aeb391" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:13.086999 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.086913 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-certificates" pod "cbd1d155-f232-4679-a70a-ac3911aeb391" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:13.088822 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.088793 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls" pod "cbd1d155-f232-4679-a70a-ac3911aeb391" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:13.092394 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.092336 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-installation-pull-secrets" pod "cbd1d155-f232-4679-a70a-ac3911aeb391" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:13.092394 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.092358 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-bound-sa-token" pod "cbd1d155-f232-4679-a70a-ac3911aeb391" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:13.092549 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.092415 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-image-registry-private-configuration" pod "cbd1d155-f232-4679-a70a-ac3911aeb391" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:13.092549 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.092446 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-kube-api-access-6qb9d" pod "cbd1d155-f232-4679-a70a-ac3911aeb391" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391"). InnerVolumeSpecName "kube-api-access-6qb9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:13.094214 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.094172 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd1d155-f232-4679-a70a-ac3911aeb391-ca-trust-extracted" pod "cbd1d155-f232-4679-a70a-ac3911aeb391" (UID: "cbd1d155-f232-4679-a70a-ac3911aeb391"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:58:13.187492 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.187413 5786 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-installation-pull-secrets\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:13.187492 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.187442 5786 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbd1d155-f232-4679-a70a-ac3911aeb391-image-registry-private-configuration\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:13.187492 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.187454 5786 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbd1d155-f232-4679-a70a-ac3911aeb391-ca-trust-extracted\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:13.187492 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.187463 5786 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-certificates\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:13.187492 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.187472 5786 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-registry-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:13.187492 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.187482 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qb9d\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-kube-api-access-6qb9d\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:13.187492 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.187490 5786 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbd1d155-f232-4679-a70a-ac3911aeb391-bound-sa-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:13.187807 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.187499 5786 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbd1d155-f232-4679-a70a-ac3911aeb391-trusted-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:13.731218 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.731183 5786 generic.go:356] "Generic (PLEG): container finished" podID="cbd1d155-f232-4679-a70a-ac3911aeb391" containerID="da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35" exitCode=0 Nov 04 00:58:13.731675 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.731232 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" Nov 04 00:58:13.731675 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.731270 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" event={"ID":"cbd1d155-f232-4679-a70a-ac3911aeb391","Type":"ContainerDied","Data":"da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35"} Nov 04 00:58:13.731675 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.731327 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5768bf545b-g67zq" event={"ID":"cbd1d155-f232-4679-a70a-ac3911aeb391","Type":"ContainerDied","Data":"2189e299abfb14d8bc685d7a52518f27aaf275387ddbc8cacafd793743f4f5ef"} Nov 04 00:58:13.731675 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.731349 5786 scope.go:122] "RemoveContainer" containerID="da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35" Nov 04 00:58:13.739746 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.739729 5786 scope.go:122] "RemoveContainer" containerID="da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35" Nov 04 00:58:13.740026 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:58:13.740007 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35\": container with ID starting with da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35 not found: ID does not exist" containerID="da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35" Nov 04 00:58:13.740076 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.740034 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35"} err="failed to get container status \"da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35\": rpc error: code = NotFound desc = could not find container \"da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35\": container with ID starting with da295143cd0959b4ef460bb073b255365d73abf1c93e931febc3b4659bf5db35 not found: ID does not exist" Nov 04 00:58:13.940609 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.940576 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5768bf545b-g67zq"] Nov 04 00:58:13.988214 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:13.988146 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5768bf545b-g67zq"] Nov 04 00:58:14.505717 ip-10-0-136-94 kubenswrapper[5786]: I1104 00:58:14.504786 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd1d155-f232-4679-a70a-ac3911aeb391" path="/var/lib/kubelet/pods/cbd1d155-f232-4679-a70a-ac3911aeb391/volumes" Nov 04 00:58:15.086926 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:58:15.086890 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 00:58:45.086275 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:58:45.086238 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 00:59:15.086680 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:59:15.086599 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 00:59:45.085594 ip-10-0-136-94 kubenswrapper[5786]: E1104 00:59:45.085553 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:00:15.085329 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:00:15.085291 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:00:45.086149 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:00:45.086045 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:01:15.086540 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:01:15.086500 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:01:45.086403 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:01:45.086364 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:02:15.087030 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:02:15.086941 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:02:34.520163 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:02:34.520135 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:02:34.521251 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:02:34.521230 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:02:45.085543 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:02:45.085498 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:03:13.867950 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.867914 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2"] Nov 04 01:03:13.868440 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.868220 5786 cpu_manager.go:414] "containerMap: removing container" podUID="cbd1d155-f232-4679-a70a-ac3911aeb391" containerName="registry" Nov 04 01:03:13.868440 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.868231 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd1d155-f232-4679-a70a-ac3911aeb391" containerName="registry" Nov 04 01:03:13.868440 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.868245 5786 cpu_manager.go:414] "containerMap: removing container" podUID="ac49addd-a050-4d12-bdc0-20345e446a15" containerName="registry" Nov 04 01:03:13.868440 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.868251 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac49addd-a050-4d12-bdc0-20345e446a15" containerName="registry" Nov 04 01:03:13.868440 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.868300 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd1d155-f232-4679-a70a-ac3911aeb391" containerName="registry" Nov 04 01:03:13.868440 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.868309 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac49addd-a050-4d12-bdc0-20345e446a15" containerName="registry" Nov 04 01:03:13.870273 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.870248 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:13.908476 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.908448 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:13.908633 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.908487 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:13.908633 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.908525 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:13.908633 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.908615 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:13.908763 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.908642 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xd4b\" (UniqueName: \"kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-kube-api-access-6xd4b\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:13.908763 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.908663 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:13.908763 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.908742 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:13.925704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.925669 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 04 01:03:13.925704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.925693 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 01:03:13.925891 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.925839 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 04 01:03:13.973199 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.973163 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7bjdx\"" Nov 04 01:03:13.973199 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.973169 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 04 01:03:13.973839 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:13.973819 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2"] Nov 04 01:03:14.009977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.009941 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.010205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.009985 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.010205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.010023 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.010205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.010054 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.010205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.010094 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xd4b\" (UniqueName: \"kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-kube-api-access-6xd4b\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.010205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.010151 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.010205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.010196 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.010870 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.010840 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.012526 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.012502 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.012526 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.012518 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.012668 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.012584 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.012712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.012689 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.068957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.068916 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.069216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.069196 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xd4b\" (UniqueName: \"kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-kube-api-access-6xd4b\") pod \"istiod-openshift-gateway-7876bfcb74-wrzh2\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.180594 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.180514 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:14.650729 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.650696 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2"] Nov 04 01:03:14.657405 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:03:14.657375 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7ad8a7_11ac_41a6_93e8_0d3fbb0856e6.slice/crio-70c5e82cbb555f02ab815035ede70b9e4f4560a8968cda2e73234ae7c868bd85 WatchSource:0}: Error finding container 70c5e82cbb555f02ab815035ede70b9e4f4560a8968cda2e73234ae7c868bd85: Status 404 returned error can't find the container with id 70c5e82cbb555f02ab815035ede70b9e4f4560a8968cda2e73234ae7c868bd85 Nov 04 01:03:14.659421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.659402 5786 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:03:14.771026 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:14.770987 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" event={"ID":"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6","Type":"ContainerStarted","Data":"70c5e82cbb555f02ab815035ede70b9e4f4560a8968cda2e73234ae7c868bd85"} Nov 04 01:03:15.086905 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:03:15.086825 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:03:18.978515 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:18.978479 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:03:18.978906 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:18.978559 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:03:19.792103 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:19.792060 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" event={"ID":"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6","Type":"ContainerStarted","Data":"ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770"} Nov 04 01:03:19.792371 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:19.792149 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:20.002843 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:20.002787 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" podStartSLOduration=2.684627809 podStartE2EDuration="7.002771921s" totalImagesPullingTime="4.318144112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:19.954539893 +0000 UTC m=+345.570885656" watchObservedRunningTime="2026-11-04 01:03:20.002771921 +0000 UTC m=+345.619117678" Nov 04 01:03:20.798077 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:20.798046 5786 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-wrzh2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:03:20.798258 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:20.798096 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" podUID="cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:03:23.798209 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:03:23.798178 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:03:45.085860 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:03:45.085774 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:04:15.086726 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:04:15.086685 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:04:16.737071 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.737038 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2"] Nov 04 01:04:16.740340 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.740322 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.802752 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.802723 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 01:04:16.805088 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.805056 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d3dbbe1-c01c-48ac-92cd-5a5821428452-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.805221 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.805096 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d3dbbe1-c01c-48ac-92cd-5a5821428452-tmp\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.805221 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.805140 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6957m\" (UniqueName: \"kubernetes.io/projected/6d3dbbe1-c01c-48ac-92cd-5a5821428452-kube-api-access-6957m\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.805292 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.805232 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d3dbbe1-c01c-48ac-92cd-5a5821428452-metrics-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.805292 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.805272 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d3dbbe1-c01c-48ac-92cd-5a5821428452-manifests\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.805353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.805291 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d3dbbe1-c01c-48ac-92cd-5a5821428452-webhook-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.839477 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.839433 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 01:04:16.839477 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.839460 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-j4czz\"" Nov 04 01:04:16.839477 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.839473 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 01:04:16.839724 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.839433 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 01:04:16.839724 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.839540 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 01:04:16.840559 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.840540 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2"] Nov 04 01:04:16.905817 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.905785 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d3dbbe1-c01c-48ac-92cd-5a5821428452-tmp\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.905989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.905828 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6957m\" (UniqueName: \"kubernetes.io/projected/6d3dbbe1-c01c-48ac-92cd-5a5821428452-kube-api-access-6957m\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.905989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.905957 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d3dbbe1-c01c-48ac-92cd-5a5821428452-metrics-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.906073 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.906017 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d3dbbe1-c01c-48ac-92cd-5a5821428452-manifests\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.906073 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.906051 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d3dbbe1-c01c-48ac-92cd-5a5821428452-webhook-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.906199 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.906115 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d3dbbe1-c01c-48ac-92cd-5a5821428452-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.906553 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.906524 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d3dbbe1-c01c-48ac-92cd-5a5821428452-manifests\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.908111 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.908085 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d3dbbe1-c01c-48ac-92cd-5a5821428452-tmp\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.908489 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.908468 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d3dbbe1-c01c-48ac-92cd-5a5821428452-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.908574 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.908516 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d3dbbe1-c01c-48ac-92cd-5a5821428452-webhook-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.908613 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.908598 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d3dbbe1-c01c-48ac-92cd-5a5821428452-metrics-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:16.963842 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:16.963804 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6957m\" (UniqueName: \"kubernetes.io/projected/6d3dbbe1-c01c-48ac-92cd-5a5821428452-kube-api-access-6957m\") pod \"opendatahub-operator-controller-manager-c655894cb-x5gf2\" (UID: \"6d3dbbe1-c01c-48ac-92cd-5a5821428452\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:17.055247 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:17.055155 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:17.706774 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:17.706739 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2"] Nov 04 01:04:17.763540 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:04:17.763511 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3dbbe1_c01c_48ac_92cd_5a5821428452.slice/crio-d7d60d3f91edf16aedce72982386fdb360613d80831598579d46a9d6e7165a75 WatchSource:0}: Error finding container d7d60d3f91edf16aedce72982386fdb360613d80831598579d46a9d6e7165a75: Status 404 returned error can't find the container with id d7d60d3f91edf16aedce72982386fdb360613d80831598579d46a9d6e7165a75 Nov 04 01:04:17.997948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:17.997870 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" event={"ID":"6d3dbbe1-c01c-48ac-92cd-5a5821428452","Type":"ContainerStarted","Data":"d7d60d3f91edf16aedce72982386fdb360613d80831598579d46a9d6e7165a75"} Nov 04 01:04:23.021074 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:23.021039 5786 generic.go:356] "Generic (PLEG): container finished" podID="6d3dbbe1-c01c-48ac-92cd-5a5821428452" containerID="c263ed4ddf1a7ba514efec357a345a74dd6d7d0c414f13592c9613cb4c2ab9f0" exitCode=0 Nov 04 01:04:23.021562 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:23.021094 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" event={"ID":"6d3dbbe1-c01c-48ac-92cd-5a5821428452","Type":"ContainerDied","Data":"c263ed4ddf1a7ba514efec357a345a74dd6d7d0c414f13592c9613cb4c2ab9f0"} Nov 04 01:04:24.029410 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:24.029374 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" event={"ID":"6d3dbbe1-c01c-48ac-92cd-5a5821428452","Type":"ContainerStarted","Data":"441c6b4336d7887ef2cb3d6f5788aad50c759ea868fc7543a4e604bee6b0a8c0"} Nov 04 01:04:24.029714 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:24.029478 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:24.306971 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:24.306876 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" podStartSLOduration=2.629192702 podStartE2EDuration="8.306863167s" totalImagesPullingTime="5.253678966s" totalInitContainerRuntime="423.991499ms" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:24.260402865 +0000 UTC m=+409.876748631" watchObservedRunningTime="2026-11-04 01:04:24.306863167 +0000 UTC m=+409.923208926" Nov 04 01:04:35.034925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:35.034886 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-x5gf2" Nov 04 01:04:45.086672 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:04:45.086634 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:04:49.931943 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:49.931913 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm"] Nov 04 01:04:49.935059 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:49.935041 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" Nov 04 01:04:50.030192 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.030156 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm"] Nov 04 01:04:50.032556 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.032536 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 01:04:50.032673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.032651 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 04 01:04:50.032730 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.032717 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-k5npg\"" Nov 04 01:04:50.075000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.074969 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1540a2-97cc-4956-836f-cdf0c8f8d810-tls-certs\") pod \"kube-auth-proxy-5794fb467c-v8xjm\" (UID: \"8b1540a2-97cc-4956-836f-cdf0c8f8d810\") " pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" Nov 04 01:04:50.075134 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.075030 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b1540a2-97cc-4956-836f-cdf0c8f8d810-tmp\") pod \"kube-auth-proxy-5794fb467c-v8xjm\" (UID: \"8b1540a2-97cc-4956-836f-cdf0c8f8d810\") " pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" Nov 04 01:04:50.075183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.075130 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bgbp\" (UniqueName: \"kubernetes.io/projected/8b1540a2-97cc-4956-836f-cdf0c8f8d810-kube-api-access-8bgbp\") pod \"kube-auth-proxy-5794fb467c-v8xjm\" (UID: \"8b1540a2-97cc-4956-836f-cdf0c8f8d810\") " pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" Nov 04 01:04:50.175987 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.175949 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1540a2-97cc-4956-836f-cdf0c8f8d810-tls-certs\") pod \"kube-auth-proxy-5794fb467c-v8xjm\" (UID: \"8b1540a2-97cc-4956-836f-cdf0c8f8d810\") " pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" Nov 04 01:04:50.176158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.176090 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b1540a2-97cc-4956-836f-cdf0c8f8d810-tmp\") pod \"kube-auth-proxy-5794fb467c-v8xjm\" (UID: \"8b1540a2-97cc-4956-836f-cdf0c8f8d810\") " pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" Nov 04 01:04:50.176158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.176110 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bgbp\" (UniqueName: \"kubernetes.io/projected/8b1540a2-97cc-4956-836f-cdf0c8f8d810-kube-api-access-8bgbp\") pod \"kube-auth-proxy-5794fb467c-v8xjm\" (UID: \"8b1540a2-97cc-4956-836f-cdf0c8f8d810\") " pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" Nov 04 01:04:50.178381 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.178350 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b1540a2-97cc-4956-836f-cdf0c8f8d810-tmp\") pod \"kube-auth-proxy-5794fb467c-v8xjm\" (UID: \"8b1540a2-97cc-4956-836f-cdf0c8f8d810\") " pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" Nov 04 01:04:50.178499 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.178470 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1540a2-97cc-4956-836f-cdf0c8f8d810-tls-certs\") pod \"kube-auth-proxy-5794fb467c-v8xjm\" (UID: \"8b1540a2-97cc-4956-836f-cdf0c8f8d810\") " pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" Nov 04 01:04:50.230607 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.230549 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bgbp\" (UniqueName: \"kubernetes.io/projected/8b1540a2-97cc-4956-836f-cdf0c8f8d810-kube-api-access-8bgbp\") pod \"kube-auth-proxy-5794fb467c-v8xjm\" (UID: \"8b1540a2-97cc-4956-836f-cdf0c8f8d810\") " pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" Nov 04 01:04:50.246339 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.246309 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" Nov 04 01:04:50.614184 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:50.614154 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm"] Nov 04 01:04:50.620790 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:04:50.620763 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1540a2_97cc_4956_836f_cdf0c8f8d810.slice/crio-d19a31cdf480ec8909004065c7d5192940c4e0b2bc34c9d7d281d6f27d11053a WatchSource:0}: Error finding container d19a31cdf480ec8909004065c7d5192940c4e0b2bc34c9d7d281d6f27d11053a: Status 404 returned error can't find the container with id d19a31cdf480ec8909004065c7d5192940c4e0b2bc34c9d7d281d6f27d11053a Nov 04 01:04:51.136999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:51.136964 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" event={"ID":"8b1540a2-97cc-4956-836f-cdf0c8f8d810","Type":"ContainerStarted","Data":"d19a31cdf480ec8909004065c7d5192940c4e0b2bc34c9d7d281d6f27d11053a"} Nov 04 01:04:54.150772 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:54.150729 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" event={"ID":"8b1540a2-97cc-4956-836f-cdf0c8f8d810","Type":"ContainerStarted","Data":"1b5ac7da6f549d1a000d1912269dc4d1b17b02a49df8ca77c3acef71704ccdb6"} Nov 04 01:04:54.260079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:04:54.260028 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5794fb467c-v8xjm" podStartSLOduration=2.108164633 podStartE2EDuration="5.260015747s" totalImagesPullingTime="3.151851114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:54.25942101 +0000 UTC m=+439.875766773" watchObservedRunningTime="2026-11-04 01:04:54.260015747 +0000 UTC m=+439.876361510" Nov 04 01:05:15.086915 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:05:15.086829 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:05:17.383823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.383784 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:05:17.387154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.387110 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" Nov 04 01:05:17.437482 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.437458 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 04 01:05:17.437634 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.437551 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 01:05:17.485232 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.485193 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qbkj\" (UniqueName: \"kubernetes.io/projected/65e12552-a3fa-4259-9b61-52b7f27d6b03-kube-api-access-8qbkj\") pod \"limitador-limitador-7c9b87d574-6j2lc\" (UID: \"65e12552-a3fa-4259-9b61-52b7f27d6b03\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" Nov 04 01:05:17.485386 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.485347 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/65e12552-a3fa-4259-9b61-52b7f27d6b03-config-file\") pod \"limitador-limitador-7c9b87d574-6j2lc\" (UID: \"65e12552-a3fa-4259-9b61-52b7f27d6b03\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" Nov 04 01:05:17.485449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.485411 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9g4nl\"" Nov 04 01:05:17.485449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.485412 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 01:05:17.486253 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.486236 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:05:17.586225 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.586190 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/65e12552-a3fa-4259-9b61-52b7f27d6b03-config-file\") pod \"limitador-limitador-7c9b87d574-6j2lc\" (UID: \"65e12552-a3fa-4259-9b61-52b7f27d6b03\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" Nov 04 01:05:17.586400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.586232 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qbkj\" (UniqueName: \"kubernetes.io/projected/65e12552-a3fa-4259-9b61-52b7f27d6b03-kube-api-access-8qbkj\") pod \"limitador-limitador-7c9b87d574-6j2lc\" (UID: \"65e12552-a3fa-4259-9b61-52b7f27d6b03\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" Nov 04 01:05:17.586844 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.586820 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/65e12552-a3fa-4259-9b61-52b7f27d6b03-config-file\") pod \"limitador-limitador-7c9b87d574-6j2lc\" (UID: \"65e12552-a3fa-4259-9b61-52b7f27d6b03\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" Nov 04 01:05:17.641627 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.641564 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qbkj\" (UniqueName: \"kubernetes.io/projected/65e12552-a3fa-4259-9b61-52b7f27d6b03-kube-api-access-8qbkj\") pod \"limitador-limitador-7c9b87d574-6j2lc\" (UID: \"65e12552-a3fa-4259-9b61-52b7f27d6b03\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" Nov 04 01:05:17.698904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.698875 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" Nov 04 01:05:17.706908 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:17.706884 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:05:18.068825 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:18.068731 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:05:18.074719 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:05:18.074690 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e12552_a3fa_4259_9b61_52b7f27d6b03.slice/crio-be3365a63a34427517f3bd0e2c99126f323da26555f1ab8eb066a7cce48cf465 WatchSource:0}: Error finding container be3365a63a34427517f3bd0e2c99126f323da26555f1ab8eb066a7cce48cf465: Status 404 returned error can't find the container with id be3365a63a34427517f3bd0e2c99126f323da26555f1ab8eb066a7cce48cf465 Nov 04 01:05:18.246768 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:18.246731 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" event={"ID":"65e12552-a3fa-4259-9b61-52b7f27d6b03","Type":"ContainerStarted","Data":"be3365a63a34427517f3bd0e2c99126f323da26555f1ab8eb066a7cce48cf465"} Nov 04 01:05:24.273864 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:24.273831 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" event={"ID":"65e12552-a3fa-4259-9b61-52b7f27d6b03","Type":"ContainerStarted","Data":"b8ad644c95b5ac64c6c5818617f38a85fdd393e3ce3cdd14b670cb9701ef1744"} Nov 04 01:05:24.274420 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:24.273906 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" Nov 04 01:05:24.385768 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:24.385714 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" podStartSLOduration=2.181493769 podStartE2EDuration="7.385700672s" totalImagesPullingTime="5.204206903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:05:24.384693931 +0000 UTC m=+470.001039693" watchObservedRunningTime="2026-11-04 01:05:24.385700672 +0000 UTC m=+470.002046435" Nov 04 01:05:35.280533 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:05:35.280502 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-6j2lc" Nov 04 01:05:45.086543 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:05:45.086509 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:06:15.086143 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:06:15.086085 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:06:21.132706 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.132672 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5"] Nov 04 01:06:21.135890 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.135866 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.176700 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.176663 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d1e41d-fded-429d-99b6-e941ffc740a6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.176873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.176708 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85d1e41d-fded-429d-99b6-e941ffc740a6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.176873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.176797 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/85d1e41d-fded-429d-99b6-e941ffc740a6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.176873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.176833 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/85d1e41d-fded-429d-99b6-e941ffc740a6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.176873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.176863 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d1e41d-fded-429d-99b6-e941ffc740a6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.177002 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.176926 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4w2h\" (UniqueName: \"kubernetes.io/projected/85d1e41d-fded-429d-99b6-e941ffc740a6-kube-api-access-m4w2h\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.177002 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.176966 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/85d1e41d-fded-429d-99b6-e941ffc740a6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.208825 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.207985 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5"] Nov 04 01:06:21.277815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.277777 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d1e41d-fded-429d-99b6-e941ffc740a6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.277996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.277828 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85d1e41d-fded-429d-99b6-e941ffc740a6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.277996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.277870 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/85d1e41d-fded-429d-99b6-e941ffc740a6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.277996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.277906 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/85d1e41d-fded-429d-99b6-e941ffc740a6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.277996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.277961 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d1e41d-fded-429d-99b6-e941ffc740a6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.277996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.277989 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4w2h\" (UniqueName: \"kubernetes.io/projected/85d1e41d-fded-429d-99b6-e941ffc740a6-kube-api-access-m4w2h\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.278280 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.278023 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/85d1e41d-fded-429d-99b6-e941ffc740a6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.278906 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.278876 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/85d1e41d-fded-429d-99b6-e941ffc740a6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.280443 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.280415 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85d1e41d-fded-429d-99b6-e941ffc740a6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.280865 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.280848 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/85d1e41d-fded-429d-99b6-e941ffc740a6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.280926 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.280869 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d1e41d-fded-429d-99b6-e941ffc740a6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.280926 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.280896 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/85d1e41d-fded-429d-99b6-e941ffc740a6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.342987 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.342952 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4w2h\" (UniqueName: \"kubernetes.io/projected/85d1e41d-fded-429d-99b6-e941ffc740a6-kube-api-access-m4w2h\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.343808 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.343778 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d1e41d-fded-429d-99b6-e941ffc740a6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cjfd5\" (UID: \"85d1e41d-fded-429d-99b6-e941ffc740a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.446919 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.446838 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:21.863628 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.863587 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5"] Nov 04 01:06:21.869878 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:06:21.869853 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d1e41d_fded_429d_99b6_e941ffc740a6.slice/crio-72849239d284ab0e853a23a2e51ed46811c1648acc9dcb25cdfd9b749155f03f WatchSource:0}: Error finding container 72849239d284ab0e853a23a2e51ed46811c1648acc9dcb25cdfd9b749155f03f: Status 404 returned error can't find the container with id 72849239d284ab0e853a23a2e51ed46811c1648acc9dcb25cdfd9b749155f03f Nov 04 01:06:21.871914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.871880 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:06:21.871973 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:21.871950 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:06:22.512765 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:22.512731 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" event={"ID":"85d1e41d-fded-429d-99b6-e941ffc740a6","Type":"ContainerStarted","Data":"09e6815e628abc7ef9422f3dd5ba8406c1a4b08adffe8094393ccf9f161dc48d"} Nov 04 01:06:22.513274 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:22.512788 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" event={"ID":"85d1e41d-fded-429d-99b6-e941ffc740a6","Type":"ContainerStarted","Data":"72849239d284ab0e853a23a2e51ed46811c1648acc9dcb25cdfd9b749155f03f"} Nov 04 01:06:22.513274 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:22.512909 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:22.514567 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:22.514542 5786 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-cjfd5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:06:22.514698 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:22.514607 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" podUID="85d1e41d-fded-429d-99b6-e941ffc740a6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:06:22.710509 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:22.710450 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" podStartSLOduration=1.710436332 podStartE2EDuration="1.710436332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:22.70960581 +0000 UTC m=+528.325951574" watchObservedRunningTime="2026-11-04 01:06:22.710436332 +0000 UTC m=+528.326782092" Nov 04 01:06:23.517516 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:23.517488 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cjfd5" Nov 04 01:06:23.727724 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:23.727689 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2"] Nov 04 01:06:23.727933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:23.727891 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" podUID="cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" containerName="discovery" containerID="cri-o://ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770" gracePeriod=30 Nov 04 01:06:23.797689 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:23.797639 5786 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-wrzh2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.25:8080/ready\": dial tcp 10.132.0.25:8080: connect: connection refused" start-of-body= Nov 04 01:06:23.797756 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:23.797702 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" podUID="cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.25:8080/ready\": dial tcp 10.132.0.25:8080: connect: connection refused" Nov 04 01:06:23.978784 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:23.978757 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:06:24.000840 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.000812 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-cacerts\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-cacerts\") pod \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " Nov 04 01:06:24.000989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.000854 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-token\" (UniqueName: \"kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-token\") pod \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " Nov 04 01:06:24.000989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.000885 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-local-certs\") pod \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " Nov 04 01:06:24.000989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.000915 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-dns-cert\") pod \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " Nov 04 01:06:24.000989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.000977 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-kube-api-access-6xd4b\" (UniqueName: \"kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-kube-api-access-6xd4b\") pod \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " Nov 04 01:06:24.001183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.000998 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-kubeconfig\") pod \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " Nov 04 01:06:24.001183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.001025 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-ca-configmap\") pod \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\" (UID: \"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6\") " Nov 04 01:06:24.001595 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.001565 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-ca-configmap" pod "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" (UID: "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:06:24.003279 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.003253 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-cacerts" pod "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" (UID: "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:06:24.003394 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.003330 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-kubeconfig" pod "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" (UID: "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:06:24.003527 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.003505 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-dns-cert" pod "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" (UID: "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:06:24.003589 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.003518 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-kube-api-access-6xd4b" pod "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" (UID: "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6"). InnerVolumeSpecName "kube-api-access-6xd4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:06:24.003636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.003620 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-local-certs" pod "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" (UID: "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:06:24.003814 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.003799 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-token" pod "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" (UID: "cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:06:24.101765 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.101737 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xd4b\" (UniqueName: \"kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-kube-api-access-6xd4b\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:24.101765 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.101765 5786 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-kubeconfig\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:24.101957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.101775 5786 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-ca-configmap\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:24.101957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.101783 5786 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-cacerts\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:24.101957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.101793 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:24.101957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.101801 5786 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-local-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:24.101957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.101810 5786 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6-istio-csr-dns-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:24.522420 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.522386 5786 generic.go:356] "Generic (PLEG): container finished" podID="cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" containerID="ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770" exitCode=0 Nov 04 01:06:24.522823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.522435 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" Nov 04 01:06:24.522823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.522469 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" event={"ID":"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6","Type":"ContainerDied","Data":"ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770"} Nov 04 01:06:24.522823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.522524 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2" event={"ID":"cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6","Type":"ContainerDied","Data":"70c5e82cbb555f02ab815035ede70b9e4f4560a8968cda2e73234ae7c868bd85"} Nov 04 01:06:24.522823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.522545 5786 scope.go:122] "RemoveContainer" containerID="ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770" Nov 04 01:06:24.530623 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.530597 5786 scope.go:122] "RemoveContainer" containerID="ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770" Nov 04 01:06:24.530931 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:06:24.530912 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770\": container with ID starting with ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770 not found: ID does not exist" containerID="ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770" Nov 04 01:06:24.530996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.530941 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770"} err="failed to get container status \"ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770\": rpc error: code = NotFound desc = could not find container \"ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770\": container with ID starting with ddee7b127657abb127f6f5d69bfb159fd1137222e24738d279d42626b4911770 not found: ID does not exist" Nov 04 01:06:24.701037 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.701002 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2"] Nov 04 01:06:24.748928 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:24.748899 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrzh2"] Nov 04 01:06:26.241866 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.241833 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mg6l2"] Nov 04 01:06:26.242289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.242154 5786 cpu_manager.go:414] "containerMap: removing container" podUID="cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" containerName="discovery" Nov 04 01:06:26.242289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.242165 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" containerName="discovery" Nov 04 01:06:26.242289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.242236 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" containerName="discovery" Nov 04 01:06:26.246565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.246545 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" Nov 04 01:06:26.299311 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.299281 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mg6l2"] Nov 04 01:06:26.317549 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.317514 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6608682-17b7-425a-bca8-55c10a066b86-tls-certs\") pod \"model-serving-api-75bd89fdfc-mg6l2\" (UID: \"d6608682-17b7-425a-bca8-55c10a066b86\") " pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" Nov 04 01:06:26.317712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.317566 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjnng\" (UniqueName: \"kubernetes.io/projected/d6608682-17b7-425a-bca8-55c10a066b86-kube-api-access-zjnng\") pod \"model-serving-api-75bd89fdfc-mg6l2\" (UID: \"d6608682-17b7-425a-bca8-55c10a066b86\") " pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" Nov 04 01:06:26.328940 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.328910 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 01:06:26.329103 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.329030 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 01:06:26.329103 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.329065 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 04 01:06:26.329103 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.329098 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-xdj86\"" Nov 04 01:06:26.348737 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.348708 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-746fj"] Nov 04 01:06:26.351889 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.351870 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:26.407190 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.407155 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-746fj"] Nov 04 01:06:26.418043 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.418010 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6608682-17b7-425a-bca8-55c10a066b86-tls-certs\") pod \"model-serving-api-75bd89fdfc-mg6l2\" (UID: \"d6608682-17b7-425a-bca8-55c10a066b86\") " pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" Nov 04 01:06:26.418232 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.418062 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78-metrics-certs\") pod \"odh-model-controller-769875db56-746fj\" (UID: \"e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78\") " pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:26.418232 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.418080 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjnng\" (UniqueName: \"kubernetes.io/projected/d6608682-17b7-425a-bca8-55c10a066b86-kube-api-access-zjnng\") pod \"model-serving-api-75bd89fdfc-mg6l2\" (UID: \"d6608682-17b7-425a-bca8-55c10a066b86\") " pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" Nov 04 01:06:26.418232 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.418102 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp8tj\" (UniqueName: \"kubernetes.io/projected/e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78-kube-api-access-wp8tj\") pod \"odh-model-controller-769875db56-746fj\" (UID: \"e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78\") " pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:26.418232 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.418142 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78-cert\") pod \"odh-model-controller-769875db56-746fj\" (UID: \"e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78\") " pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:26.420571 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.420543 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6608682-17b7-425a-bca8-55c10a066b86-tls-certs\") pod \"model-serving-api-75bd89fdfc-mg6l2\" (UID: \"d6608682-17b7-425a-bca8-55c10a066b86\") " pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" Nov 04 01:06:26.423689 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.423667 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 04 01:06:26.423689 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.423669 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 04 01:06:26.423814 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.423670 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qldpm\"" Nov 04 01:06:26.472333 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.472292 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjnng\" (UniqueName: \"kubernetes.io/projected/d6608682-17b7-425a-bca8-55c10a066b86-kube-api-access-zjnng\") pod \"model-serving-api-75bd89fdfc-mg6l2\" (UID: \"d6608682-17b7-425a-bca8-55c10a066b86\") " pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" Nov 04 01:06:26.503273 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.503196 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6" path="/var/lib/kubelet/pods/cc7ad8a7-11ac-41a6-93e8-0d3fbb0856e6/volumes" Nov 04 01:06:26.518772 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.518736 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp8tj\" (UniqueName: \"kubernetes.io/projected/e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78-kube-api-access-wp8tj\") pod \"odh-model-controller-769875db56-746fj\" (UID: \"e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78\") " pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:26.518916 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.518782 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78-cert\") pod \"odh-model-controller-769875db56-746fj\" (UID: \"e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78\") " pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:26.518916 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.518845 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78-metrics-certs\") pod \"odh-model-controller-769875db56-746fj\" (UID: \"e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78\") " pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:26.521327 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.521307 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78-metrics-certs\") pod \"odh-model-controller-769875db56-746fj\" (UID: \"e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78\") " pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:26.521398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.521344 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78-cert\") pod \"odh-model-controller-769875db56-746fj\" (UID: \"e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78\") " pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:26.558201 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.558163 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" Nov 04 01:06:26.574259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.574231 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp8tj\" (UniqueName: \"kubernetes.io/projected/e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78-kube-api-access-wp8tj\") pod \"odh-model-controller-769875db56-746fj\" (UID: \"e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78\") " pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:26.665314 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.665279 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:26.925427 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:26.925398 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mg6l2"] Nov 04 01:06:26.932338 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:06:26.932309 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6608682_17b7_425a_bca8_55c10a066b86.slice/crio-075c660600ad6b26cdf7c7a96b5dd0997e911230787f3b8a45b7e6eaa698f327 WatchSource:0}: Error finding container 075c660600ad6b26cdf7c7a96b5dd0997e911230787f3b8a45b7e6eaa698f327: Status 404 returned error can't find the container with id 075c660600ad6b26cdf7c7a96b5dd0997e911230787f3b8a45b7e6eaa698f327 Nov 04 01:06:27.039314 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:27.039280 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-746fj"] Nov 04 01:06:27.044776 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:06:27.044747 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bfe1fc_5272_43b7_ba5b_e57f2c8e9d78.slice/crio-da99bb155cc329b4cca858fac1648c9894d7b89bf851187d4686937ab007fffc WatchSource:0}: Error finding container da99bb155cc329b4cca858fac1648c9894d7b89bf851187d4686937ab007fffc: Status 404 returned error can't find the container with id da99bb155cc329b4cca858fac1648c9894d7b89bf851187d4686937ab007fffc Nov 04 01:06:27.537876 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:27.537837 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-746fj" event={"ID":"e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78","Type":"ContainerStarted","Data":"da99bb155cc329b4cca858fac1648c9894d7b89bf851187d4686937ab007fffc"} Nov 04 01:06:27.539046 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:27.539019 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" event={"ID":"d6608682-17b7-425a-bca8-55c10a066b86","Type":"ContainerStarted","Data":"075c660600ad6b26cdf7c7a96b5dd0997e911230787f3b8a45b7e6eaa698f327"} Nov 04 01:06:28.822702 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:28.822663 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:06:30.556224 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:30.556176 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" event={"ID":"d6608682-17b7-425a-bca8-55c10a066b86","Type":"ContainerStarted","Data":"196647fc654f67a28e402fae6c68ff38f2ff59571d6f858a6b91a9f81d6e56d6"} Nov 04 01:06:30.556714 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:30.556343 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" Nov 04 01:06:31.562394 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:31.562359 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-746fj" event={"ID":"e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78","Type":"ContainerStarted","Data":"2398d712ecca570aeb67c6f64ccdc3eb7ea265ebcc43d9ae4472eb3ba750fab2"} Nov 04 01:06:31.562794 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:31.562566 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:31.671317 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:31.671260 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" podStartSLOduration=2.283131288 podStartE2EDuration="5.671247219s" totalImagesPullingTime="3.388115931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:30.668989586 +0000 UTC m=+536.285335350" watchObservedRunningTime="2026-11-04 01:06:31.671247219 +0000 UTC m=+537.287592976" Nov 04 01:06:31.671930 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:31.671903 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-746fj" podStartSLOduration=1.836914132 podStartE2EDuration="5.671897257s" totalImagesPullingTime="3.834983125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:31.671093828 +0000 UTC m=+537.287439590" watchObservedRunningTime="2026-11-04 01:06:31.671897257 +0000 UTC m=+537.288243407" Nov 04 01:06:33.713099 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.713065 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2"] Nov 04 01:06:33.716600 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.716580 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:33.785210 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.785178 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85d60f5-b598-4909-bffd-b383252a2be5-openshift-service-ca\") pod \"maas-api-7894895fc6-q8jq2\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:33.785374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.785242 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc2k5\" (UniqueName: \"kubernetes.io/projected/d85d60f5-b598-4909-bffd-b383252a2be5-kube-api-access-zc2k5\") pod \"maas-api-7894895fc6-q8jq2\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:33.785424 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.785382 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-metrics-certs\") pod \"maas-api-7894895fc6-q8jq2\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:33.785461 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.785425 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-maas-api-tls\") pod \"maas-api-7894895fc6-q8jq2\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:33.814646 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.814608 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8zmjb\"" Nov 04 01:06:33.814646 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.814623 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 04 01:06:33.814851 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.814650 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:06:33.814907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.814846 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:06:33.814907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.814873 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:06:33.815916 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.815899 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2"] Nov 04 01:06:33.886670 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.886634 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85d60f5-b598-4909-bffd-b383252a2be5-openshift-service-ca\") pod \"maas-api-7894895fc6-q8jq2\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:33.886865 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.886689 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc2k5\" (UniqueName: \"kubernetes.io/projected/d85d60f5-b598-4909-bffd-b383252a2be5-kube-api-access-zc2k5\") pod \"maas-api-7894895fc6-q8jq2\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:33.886865 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.886747 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-metrics-certs\") pod \"maas-api-7894895fc6-q8jq2\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:33.886865 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.886797 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-maas-api-tls\") pod \"maas-api-7894895fc6-q8jq2\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:33.887468 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.887442 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85d60f5-b598-4909-bffd-b383252a2be5-openshift-service-ca\") pod \"maas-api-7894895fc6-q8jq2\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:33.889246 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.889221 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-metrics-certs\") pod \"maas-api-7894895fc6-q8jq2\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:33.889382 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.889361 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-maas-api-tls\") pod \"maas-api-7894895fc6-q8jq2\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:33.940674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:33.940637 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc2k5\" (UniqueName: \"kubernetes.io/projected/d85d60f5-b598-4909-bffd-b383252a2be5-kube-api-access-zc2k5\") pod \"maas-api-7894895fc6-q8jq2\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:34.028780 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:34.028697 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:34.395779 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:34.395744 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2"] Nov 04 01:06:34.401876 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:06:34.401846 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85d60f5_b598_4909_bffd_b383252a2be5.slice/crio-8f94f7e40973d534012d60db7bd0f4c9e961f1b679f0382617f0fe75fded8327 WatchSource:0}: Error finding container 8f94f7e40973d534012d60db7bd0f4c9e961f1b679f0382617f0fe75fded8327: Status 404 returned error can't find the container with id 8f94f7e40973d534012d60db7bd0f4c9e961f1b679f0382617f0fe75fded8327 Nov 04 01:06:34.575008 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:34.574975 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" event={"ID":"d85d60f5-b598-4909-bffd-b383252a2be5","Type":"ContainerStarted","Data":"8f94f7e40973d534012d60db7bd0f4c9e961f1b679f0382617f0fe75fded8327"} Nov 04 01:06:38.594428 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:38.594391 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" event={"ID":"d85d60f5-b598-4909-bffd-b383252a2be5","Type":"ContainerStarted","Data":"ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88"} Nov 04 01:06:38.594828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:38.594508 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:38.704463 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:38.704402 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" podStartSLOduration=1.764475774 podStartE2EDuration="5.704388636s" totalImagesPullingTime="3.939912862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:38.703313891 +0000 UTC m=+544.319659653" watchObservedRunningTime="2026-11-04 01:06:38.704388636 +0000 UTC m=+544.320734398" Nov 04 01:06:41.566460 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:41.566432 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-mg6l2" Nov 04 01:06:42.569192 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:42.569162 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-746fj" Nov 04 01:06:44.604106 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:06:44.604078 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:06:45.085750 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:06:45.085668 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:07:15.086598 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:07:15.086552 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:07:34.545396 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:07:34.545366 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:07:34.547085 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:07:34.547066 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:07:45.086902 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:07:45.086861 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:08:15.086791 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:08:15.086713 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:08:39.607934 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:39.607899 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:08:44.582253 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.582219 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2"] Nov 04 01:08:44.584517 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.584499 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.666128 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.666083 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.666281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.666143 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbwc\" (UniqueName: \"kubernetes.io/projected/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-kube-api-access-6sbwc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.666281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.666201 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.666281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.666252 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.666413 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.666332 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.666413 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.666371 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.687096 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.687061 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 01:08:44.687281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.687181 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 01:08:44.687281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.687223 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 04 01:08:44.687424 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.687342 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hdglk\"" Nov 04 01:08:44.688424 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.688404 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2"] Nov 04 01:08:44.767638 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.767592 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.767886 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.767751 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.767886 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.767822 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.767886 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.767855 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.768048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.767895 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.768048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.767935 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbwc\" (UniqueName: \"kubernetes.io/projected/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-kube-api-access-6sbwc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.768048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.768013 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.768234 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.768072 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.768295 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.768259 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.772484 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.772463 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.772790 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.772769 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.824579 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.824544 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbwc\" (UniqueName: \"kubernetes.io/projected/0b8dfc4b-7e71-4360-91cd-a22c7685da9e-kube-api-access-6sbwc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2\" (UID: \"0b8dfc4b-7e71-4360-91cd-a22c7685da9e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:44.913045 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:44.913004 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:45.086779 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:08:45.086747 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:08:45.281115 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:45.281036 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2"] Nov 04 01:08:45.287737 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:08:45.287699 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b8dfc4b_7e71_4360_91cd_a22c7685da9e.slice/crio-f7a4d53e403d0d4f2b4737a06316f0e946e8d3a8a040039c512eb51ea84056bf WatchSource:0}: Error finding container f7a4d53e403d0d4f2b4737a06316f0e946e8d3a8a040039c512eb51ea84056bf: Status 404 returned error can't find the container with id f7a4d53e403d0d4f2b4737a06316f0e946e8d3a8a040039c512eb51ea84056bf Nov 04 01:08:45.288942 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:45.288924 5786 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:08:46.106776 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:46.106734 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" event={"ID":"0b8dfc4b-7e71-4360-91cd-a22c7685da9e","Type":"ContainerStarted","Data":"f7a4d53e403d0d4f2b4737a06316f0e946e8d3a8a040039c512eb51ea84056bf"} Nov 04 01:08:47.112430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:47.112389 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" event={"ID":"0b8dfc4b-7e71-4360-91cd-a22c7685da9e","Type":"ContainerStarted","Data":"ad762fef470672b276197448fd605df6dff5006a97ec98e549d8d0eb9ea2ac8c"} Nov 04 01:08:47.237836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:47.237784 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podStartSLOduration=1.752204729 podStartE2EDuration="3.237772214s" totalImagesPullingTime="1.485567485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:47.236784774 +0000 UTC m=+672.853130539" watchObservedRunningTime="2026-11-04 01:08:47.237772214 +0000 UTC m=+672.854117975" Nov 04 01:08:47.914168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:47.914111 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:52.113467 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.113435 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd"] Nov 04 01:08:52.116896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.116877 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.214757 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.214729 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 04 01:08:52.216077 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.216056 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd"] Nov 04 01:08:52.227935 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.227914 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a107e-19f3-4611-90ad-f345b54e27f5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.228031 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.227946 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a5a107e-19f3-4611-90ad-f345b54e27f5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.228031 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.227971 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a5a107e-19f3-4611-90ad-f345b54e27f5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.228140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.228098 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a5a107e-19f3-4611-90ad-f345b54e27f5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.228187 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.228168 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5a107e-19f3-4611-90ad-f345b54e27f5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.228232 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.228220 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82xcr\" (UniqueName: \"kubernetes.io/projected/3a5a107e-19f3-4611-90ad-f345b54e27f5-kube-api-access-82xcr\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.329392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.329360 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82xcr\" (UniqueName: \"kubernetes.io/projected/3a5a107e-19f3-4611-90ad-f345b54e27f5-kube-api-access-82xcr\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.329392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.329396 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a107e-19f3-4611-90ad-f345b54e27f5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.329631 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.329417 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a5a107e-19f3-4611-90ad-f345b54e27f5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.329631 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.329444 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a5a107e-19f3-4611-90ad-f345b54e27f5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.329631 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.329481 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a5a107e-19f3-4611-90ad-f345b54e27f5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.329631 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.329513 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5a107e-19f3-4611-90ad-f345b54e27f5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.329887 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.329838 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a107e-19f3-4611-90ad-f345b54e27f5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.329887 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.329877 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a5a107e-19f3-4611-90ad-f345b54e27f5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.329962 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.329928 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a5a107e-19f3-4611-90ad-f345b54e27f5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.331823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.331787 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a5a107e-19f3-4611-90ad-f345b54e27f5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.332009 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.331984 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5a107e-19f3-4611-90ad-f345b54e27f5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.384330 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.384266 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82xcr\" (UniqueName: \"kubernetes.io/projected/3a5a107e-19f3-4611-90ad-f345b54e27f5-kube-api-access-82xcr\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-85cpd\" (UID: \"3a5a107e-19f3-4611-90ad-f345b54e27f5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.429252 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.429221 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:52.794069 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:52.793987 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd"] Nov 04 01:08:52.800338 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:08:52.800310 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5a107e_19f3_4611_90ad_f345b54e27f5.slice/crio-c5d2918fd5b4b72477f499c547d3d71ebca34c44636b18b3f129a82186016c73 WatchSource:0}: Error finding container c5d2918fd5b4b72477f499c547d3d71ebca34c44636b18b3f129a82186016c73: Status 404 returned error can't find the container with id c5d2918fd5b4b72477f499c547d3d71ebca34c44636b18b3f129a82186016c73 Nov 04 01:08:53.138002 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.137970 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" event={"ID":"3a5a107e-19f3-4611-90ad-f345b54e27f5","Type":"ContainerStarted","Data":"c5d2918fd5b4b72477f499c547d3d71ebca34c44636b18b3f129a82186016c73"} Nov 04 01:08:53.858561 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.858531 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw"] Nov 04 01:08:53.861972 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.861954 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:53.941743 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.941698 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:53.941743 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.941744 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:53.941946 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.941768 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxddl\" (UniqueName: \"kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-kube-api-access-qxddl\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:53.941946 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.941897 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad79149-b0b3-4a22-a116-8bc3ca56f325-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:53.942017 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.941960 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:53.942059 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.942035 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:53.942091 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.942069 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:53.942169 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.942104 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:53.942235 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.942194 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:53.959677 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.959653 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fwrps\"" Nov 04 01:08:53.961531 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:53.961499 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw"] Nov 04 01:08:54.043269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043241 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.043269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043272 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.043565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043305 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.043565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043323 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.043565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043340 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxddl\" (UniqueName: \"kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-kube-api-access-qxddl\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.043565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043362 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad79149-b0b3-4a22-a116-8bc3ca56f325-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.043565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043393 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.043565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043459 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.043565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043489 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.043935 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043700 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.043935 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043732 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.043935 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043807 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.044084 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.043968 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.044175 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.044155 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad79149-b0b3-4a22-a116-8bc3ca56f325-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.045647 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.045628 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.045985 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.045961 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.098323 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.098280 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.098477 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.098411 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxddl\" (UniqueName: \"kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-kube-api-access-qxddl\") pod \"maas-default-gateway-openshift-default-694db55c87-mrdrw\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.143169 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.143069 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" event={"ID":"3a5a107e-19f3-4611-90ad-f345b54e27f5","Type":"ContainerStarted","Data":"100c7a9338d57169cb9d4a4fc744362a47d2aa7bac950d467932692ad0b0e56c"} Nov 04 01:08:54.174249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.174220 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:08:54.253540 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.253490 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" podStartSLOduration=1.678001362 podStartE2EDuration="2.253475296s" totalImagesPullingTime="575.473934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:54.253254728 +0000 UTC m=+679.869600491" watchObservedRunningTime="2026-11-04 01:08:54.253475296 +0000 UTC m=+679.869821060" Nov 04 01:08:54.429412 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.429329 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:08:54.600488 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.600456 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw"] Nov 04 01:08:54.607946 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:08:54.607913 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad79149_b0b3_4a22_a116_8bc3ca56f325.slice/crio-3323d0654acbdf9cbaa5e45db915ac2f2c78dfabb4815031957008c074327052 WatchSource:0}: Error finding container 3323d0654acbdf9cbaa5e45db915ac2f2c78dfabb4815031957008c074327052: Status 404 returned error can't find the container with id 3323d0654acbdf9cbaa5e45db915ac2f2c78dfabb4815031957008c074327052 Nov 04 01:08:54.913309 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.913271 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:54.925746 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:54.925716 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:08:55.052038 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:55.052008 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:08:55.148712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:55.148671 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" event={"ID":"fad79149-b0b3-4a22-a116-8bc3ca56f325","Type":"ContainerStarted","Data":"3323d0654acbdf9cbaa5e45db915ac2f2c78dfabb4815031957008c074327052"} Nov 04 01:08:55.160458 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:08:55.160425 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:09:00.068209 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.068177 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:09:00.068476 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.068248 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:09:00.068476 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.068276 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:09:00.173949 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.173913 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" event={"ID":"fad79149-b0b3-4a22-a116-8bc3ca56f325","Type":"ContainerStarted","Data":"fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e"} Nov 04 01:09:00.174456 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.174434 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:09:00.204181 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.204142 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk"] Nov 04 01:09:00.207659 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.207632 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.271745 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.271718 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 04 01:09:00.272822 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.272800 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk"] Nov 04 01:09:00.296066 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.296026 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5701df54-b91d-44ca-b43c-de2454ab59d8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.296273 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.296199 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5701df54-b91d-44ca-b43c-de2454ab59d8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.296273 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.296238 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5701df54-b91d-44ca-b43c-de2454ab59d8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.296402 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.296309 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5701df54-b91d-44ca-b43c-de2454ab59d8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.296483 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.296462 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5701df54-b91d-44ca-b43c-de2454ab59d8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.296542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.296495 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5g2r\" (UniqueName: \"kubernetes.io/projected/5701df54-b91d-44ca-b43c-de2454ab59d8-kube-api-access-x5g2r\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.314739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.314692 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" podStartSLOduration=1.857519338 podStartE2EDuration="7.314680149s" totalImagesPullingTime="5.457160811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:00.314185371 +0000 UTC m=+685.930531133" watchObservedRunningTime="2026-11-04 01:09:00.314680149 +0000 UTC m=+685.931025912" Nov 04 01:09:00.397880 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.397846 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5701df54-b91d-44ca-b43c-de2454ab59d8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.398056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.397890 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5701df54-b91d-44ca-b43c-de2454ab59d8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.398056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.397911 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5g2r\" (UniqueName: \"kubernetes.io/projected/5701df54-b91d-44ca-b43c-de2454ab59d8-kube-api-access-x5g2r\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.398056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.397929 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5701df54-b91d-44ca-b43c-de2454ab59d8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.398056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.397981 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5701df54-b91d-44ca-b43c-de2454ab59d8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.398056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.397994 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5701df54-b91d-44ca-b43c-de2454ab59d8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.398386 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.398361 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5701df54-b91d-44ca-b43c-de2454ab59d8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.398438 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.398405 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5701df54-b91d-44ca-b43c-de2454ab59d8-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.398478 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.398436 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5701df54-b91d-44ca-b43c-de2454ab59d8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.400228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.400203 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5701df54-b91d-44ca-b43c-de2454ab59d8-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.400491 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.400474 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5701df54-b91d-44ca-b43c-de2454ab59d8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.453312 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.453278 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5g2r\" (UniqueName: \"kubernetes.io/projected/5701df54-b91d-44ca-b43c-de2454ab59d8-kube-api-access-x5g2r\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hz9wk\" (UID: \"5701df54-b91d-44ca-b43c-de2454ab59d8\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.521762 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.521723 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:00.918573 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:00.918538 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk"] Nov 04 01:09:00.924534 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:09:00.924503 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5701df54_b91d_44ca_b43c_de2454ab59d8.slice/crio-d9162822dd560f15f23587fc8a0288bca1b8c6a12ce6c645fd79ae2c21ad1f73 WatchSource:0}: Error finding container d9162822dd560f15f23587fc8a0288bca1b8c6a12ce6c645fd79ae2c21ad1f73: Status 404 returned error can't find the container with id d9162822dd560f15f23587fc8a0288bca1b8c6a12ce6c645fd79ae2c21ad1f73 Nov 04 01:09:01.185992 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:01.185576 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mrdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:09:01.185992 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:01.185641 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 04 01:09:01.191048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:01.191017 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" event={"ID":"5701df54-b91d-44ca-b43c-de2454ab59d8","Type":"ContainerStarted","Data":"d9162822dd560f15f23587fc8a0288bca1b8c6a12ce6c645fd79ae2c21ad1f73"} Nov 04 01:09:02.186577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:02.186538 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mrdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:09:02.186974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:02.186604 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 04 01:09:02.197102 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:02.197066 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" event={"ID":"5701df54-b91d-44ca-b43c-de2454ab59d8","Type":"ContainerStarted","Data":"3721c70e491a903d76e2996846095136e5eb01e9ee9bf14856666206821ed514"} Nov 04 01:09:02.309973 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:02.309904 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" podStartSLOduration=1.723630336 podStartE2EDuration="2.309890823s" totalImagesPullingTime="586.260487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:02.308755175 +0000 UTC m=+687.925100939" watchObservedRunningTime="2026-11-04 01:09:02.309890823 +0000 UTC m=+687.926236597" Nov 04 01:09:02.429747 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:02.429713 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:09:02.442391 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:02.442327 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:09:02.522445 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:02.522413 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:03.186254 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:03.186213 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mrdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:09:03.186488 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:03.186285 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 04 01:09:03.212199 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:03.212167 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" Nov 04 01:09:04.185143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.184552 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mrdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:09:04.185143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.184672 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 04 01:09:04.587595 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.587510 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k"] Nov 04 01:09:04.591609 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.591587 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.635310 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.635274 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdtkg\" (UniqueName: \"kubernetes.io/projected/d0506411-64e5-47a9-b8b2-d03f78db8d67-kube-api-access-gdtkg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.635488 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.635321 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0506411-64e5-47a9-b8b2-d03f78db8d67-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.635488 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.635344 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0506411-64e5-47a9-b8b2-d03f78db8d67-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.635488 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.635369 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0506411-64e5-47a9-b8b2-d03f78db8d67-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.635488 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.635387 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0506411-64e5-47a9-b8b2-d03f78db8d67-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.635488 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.635441 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0506411-64e5-47a9-b8b2-d03f78db8d67-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.644648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.644617 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 04 01:09:04.693884 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.693852 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k"] Nov 04 01:09:04.736132 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.736078 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0506411-64e5-47a9-b8b2-d03f78db8d67-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.736315 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.736146 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0506411-64e5-47a9-b8b2-d03f78db8d67-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.736315 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.736186 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0506411-64e5-47a9-b8b2-d03f78db8d67-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.736315 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.736260 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0506411-64e5-47a9-b8b2-d03f78db8d67-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.736438 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.736374 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0506411-64e5-47a9-b8b2-d03f78db8d67-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.736438 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.736417 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdtkg\" (UniqueName: \"kubernetes.io/projected/d0506411-64e5-47a9-b8b2-d03f78db8d67-kube-api-access-gdtkg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.736638 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.736616 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0506411-64e5-47a9-b8b2-d03f78db8d67-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.736925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.736905 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0506411-64e5-47a9-b8b2-d03f78db8d67-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.736996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.736965 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0506411-64e5-47a9-b8b2-d03f78db8d67-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.739107 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.739085 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0506411-64e5-47a9-b8b2-d03f78db8d67-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.739712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.739694 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0506411-64e5-47a9-b8b2-d03f78db8d67-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.796654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.796619 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdtkg\" (UniqueName: \"kubernetes.io/projected/d0506411-64e5-47a9-b8b2-d03f78db8d67-kube-api-access-gdtkg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k\" (UID: \"d0506411-64e5-47a9-b8b2-d03f78db8d67\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:04.904621 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:04.904571 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:05.187101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:05.186586 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mrdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:09:05.187101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:05.186662 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 04 01:09:05.295316 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:05.295281 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k"] Nov 04 01:09:05.302160 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:09:05.301917 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0506411_64e5_47a9_b8b2_d03f78db8d67.slice/crio-2f5f8077e02d4b5d475fd61dd9121dc431c779542f2f20cb2e681f76bd1e135c WatchSource:0}: Error finding container 2f5f8077e02d4b5d475fd61dd9121dc431c779542f2f20cb2e681f76bd1e135c: Status 404 returned error can't find the container with id 2f5f8077e02d4b5d475fd61dd9121dc431c779542f2f20cb2e681f76bd1e135c Nov 04 01:09:06.175458 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:06.175423 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mrdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:09:06.175843 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:06.175487 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 04 01:09:06.215762 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:06.215679 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" event={"ID":"d0506411-64e5-47a9-b8b2-d03f78db8d67","Type":"ContainerStarted","Data":"21f3758a9a54b6d5f30b716ca82d80eec9325e95af0af144b89ad036eee17ff4"} Nov 04 01:09:06.215762 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:06.215728 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" event={"ID":"d0506411-64e5-47a9-b8b2-d03f78db8d67","Type":"ContainerStarted","Data":"2f5f8077e02d4b5d475fd61dd9121dc431c779542f2f20cb2e681f76bd1e135c"} Nov 04 01:09:06.337267 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:06.337204 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" podStartSLOduration=1.704082767 podStartE2EDuration="2.337185052s" totalImagesPullingTime="633.102285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:06.335608446 +0000 UTC m=+691.951954207" watchObservedRunningTime="2026-11-04 01:09:06.337185052 +0000 UTC m=+691.953530819" Nov 04 01:09:06.905473 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:06.905433 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:08.175617 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:08.175577 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mrdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:09:08.176057 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:08.175635 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:09:09.175040 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:09.174997 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:09:09.216261 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:09.216224 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mrdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:09:09.216790 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:09.216280 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:09:09.294644 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:09.294609 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:09:09.296009 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:09.295987 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:09:10.521857 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:10.521812 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:10.534389 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:10.534365 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:11.248415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:11.248388 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hz9wk" Nov 04 01:09:12.104658 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:12.104625 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:09:14.905217 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:14.905179 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:14.917796 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:14.917771 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:15.087297 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:09:15.087253 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:09:15.267047 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:15.266975 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" Nov 04 01:09:39.239471 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.239439 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx"] Nov 04 01:09:39.246312 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.246288 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.345294 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.345251 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx"] Nov 04 01:09:39.345765 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.345737 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.345930 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.345796 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.345987 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.345926 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.345987 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.345976 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.346099 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.346028 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.346376 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.346348 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.346462 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.346406 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmsxn\" (UniqueName: \"kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-kube-api-access-pmsxn\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.346730 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.346704 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.346861 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.346750 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98e80b6-dee1-4f43-a02f-c9becae937bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.447414 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.447374 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.447583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.447423 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmsxn\" (UniqueName: \"kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-kube-api-access-pmsxn\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.447583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.447465 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.447583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.447500 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98e80b6-dee1-4f43-a02f-c9becae937bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.447583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.447546 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.447583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.447567 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.447843 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.447598 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.447843 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.447630 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.447843 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.447675 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.447999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.447895 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.448056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.447995 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.448181 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.448153 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.448327 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.448311 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98e80b6-dee1-4f43-a02f-c9becae937bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.448392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.448361 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.449925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.449908 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.450052 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.450035 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.502689 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.502618 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.502825 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.502804 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmsxn\" (UniqueName: \"kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-kube-api-access-pmsxn\") pod \"maas-default-gateway-openshift-default-694db55c87-dk4wx\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.561353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.561321 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:39.981465 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.981432 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx"] Nov 04 01:09:39.987639 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:09:39.987615 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98e80b6_dee1_4f43_a02f_c9becae937bd.slice/crio-6652e37a14a1097cd0d6f6bbfbb5e6ef660422fc32c2c34ae9df7f0e3c373826 WatchSource:0}: Error finding container 6652e37a14a1097cd0d6f6bbfbb5e6ef660422fc32c2c34ae9df7f0e3c373826: Status 404 returned error can't find the container with id 6652e37a14a1097cd0d6f6bbfbb5e6ef660422fc32c2c34ae9df7f0e3c373826 Nov 04 01:09:39.989575 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.989537 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:09:39.989671 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.989617 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:09:39.989671 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:39.989662 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:09:40.366567 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:40.366529 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" event={"ID":"b98e80b6-dee1-4f43-a02f-c9becae937bd","Type":"ContainerStarted","Data":"ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18"} Nov 04 01:09:40.366978 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:40.366594 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" event={"ID":"b98e80b6-dee1-4f43-a02f-c9becae937bd","Type":"ContainerStarted","Data":"6652e37a14a1097cd0d6f6bbfbb5e6ef660422fc32c2c34ae9df7f0e3c373826"} Nov 04 01:09:40.485941 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:40.485874 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" podStartSLOduration=1.485855087 podStartE2EDuration="1.485855087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:40.485608181 +0000 UTC m=+726.101953945" watchObservedRunningTime="2026-11-04 01:09:40.485855087 +0000 UTC m=+726.102200853" Nov 04 01:09:40.581276 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:40.581218 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:41.579144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:41.571965 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dk4wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 04 01:09:41.579144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:41.572027 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" podUID="b98e80b6-dee1-4f43-a02f-c9becae937bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 04 01:09:42.580424 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:42.580388 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dk4wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 04 01:09:42.580986 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:42.580955 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" podUID="b98e80b6-dee1-4f43-a02f-c9becae937bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 04 01:09:43.582143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:43.576526 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dk4wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 04 01:09:43.582143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:43.576590 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" podUID="b98e80b6-dee1-4f43-a02f-c9becae937bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 04 01:09:44.586332 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:44.586236 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dk4wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 04 01:09:44.586332 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:44.586299 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" podUID="b98e80b6-dee1-4f43-a02f-c9becae937bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 04 01:09:45.109671 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:09:45.109632 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:09:45.578207 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:45.576604 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dk4wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 04 01:09:45.578207 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:45.576690 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" podUID="b98e80b6-dee1-4f43-a02f-c9becae937bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 04 01:09:46.562470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:46.562431 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dk4wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 04 01:09:46.562894 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:46.562505 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" podUID="b98e80b6-dee1-4f43-a02f-c9becae937bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 04 01:09:48.213672 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:48.213638 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:48.214193 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:48.214032 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:09:48.214833 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:09:48.214812 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:10:15.086866 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:10:15.086827 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:10:43.791419 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.791377 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2"] Nov 04 01:10:43.795386 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.795359 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.853368 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.852992 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2"] Nov 04 01:10:43.853538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.853417 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.853538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.853462 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.853538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.853488 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.853701 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.853561 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.853701 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.853656 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.853784 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.853702 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jx6g\" (UniqueName: \"kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-kube-api-access-5jx6g\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.853784 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.853736 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.853892 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.853863 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.853947 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.853910 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.865545 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.865522 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-678aaf-openshift-default-dockercfg-88gm8\"" Nov 04 01:10:43.955182 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.955099 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.955372 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.955192 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.955372 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.955218 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.955372 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.955246 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.955523 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.955439 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.955580 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.955528 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jx6g\" (UniqueName: \"kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-kube-api-access-5jx6g\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.955641 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.955607 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.955697 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.955662 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.955697 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.955674 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.955804 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.955708 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.955804 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.955722 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.956064 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.956041 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.956180 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.956053 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.956228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.956181 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.958031 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.958006 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:43.958458 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:43.958435 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:44.010135 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.010097 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:44.010284 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.010205 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jx6g\" (UniqueName: \"kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-kube-api-access-5jx6g\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:44.107914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.107882 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:44.455992 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.455915 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g"] Nov 04 01:10:44.460379 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.460351 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.510545 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.510509 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e993d7-openshift-default-dockercfg-fr6p4\"" Nov 04 01:10:44.517538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.517489 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g"] Nov 04 01:10:44.562517 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.562480 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.562682 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.562527 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.562682 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.562548 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.562682 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.562572 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.562682 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.562644 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6rnk\" (UniqueName: \"kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-kube-api-access-l6rnk\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.562816 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.562684 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.562816 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.562706 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.562877 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.562834 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febf58ae-2630-40f9-a463-be0d0c583f2b-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.562877 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.562865 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.583538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.583511 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2"] Nov 04 01:10:44.590341 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:10:44.590306 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09a5559_a4e6_41f4_9dca_1806e1f61a8e.slice/crio-a964388191b20c96834d845b41f7f4e2e0d581ac280bb79c86bbc62fbaf655fa WatchSource:0}: Error finding container a964388191b20c96834d845b41f7f4e2e0d581ac280bb79c86bbc62fbaf655fa: Status 404 returned error can't find the container with id a964388191b20c96834d845b41f7f4e2e0d581ac280bb79c86bbc62fbaf655fa Nov 04 01:10:44.592606 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.592574 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:10:44.592694 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.592640 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:10:44.592694 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.592666 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:10:44.663748 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.663715 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.663909 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.663770 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6rnk\" (UniqueName: \"kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-kube-api-access-l6rnk\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.664158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.664101 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.664237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.664184 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.664237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.664214 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febf58ae-2630-40f9-a463-be0d0c583f2b-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.664326 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.664257 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.664407 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.664388 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.664485 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.664468 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.664530 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.664506 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.664937 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.664916 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.665468 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.665444 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.665709 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.665688 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.666538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.666253 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febf58ae-2630-40f9-a463-be0d0c583f2b-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.666538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.666501 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.668643 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.668611 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" event={"ID":"a09a5559-a4e6-41f4-9dca-1806e1f61a8e","Type":"ContainerStarted","Data":"a964388191b20c96834d845b41f7f4e2e0d581ac280bb79c86bbc62fbaf655fa"} Nov 04 01:10:44.668781 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.668760 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.670312 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.670295 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.724513 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.724422 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.724513 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.724499 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6rnk\" (UniqueName: \"kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-kube-api-access-l6rnk\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:44.774449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:44.774417 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:45.087387 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:10:45.087307 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:10:45.147634 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.147601 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4"] Nov 04 01:10:45.152292 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.152269 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.196845 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.196814 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g"] Nov 04 01:10:45.203204 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:10:45.203170 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebf58ae_2630_40f9_a463_be0d0c583f2b.slice/crio-e7971be82f38123c9e9aaceefc4c597df4dd898dd89ac4620b6953d96856d496 WatchSource:0}: Error finding container e7971be82f38123c9e9aaceefc4c597df4dd898dd89ac4620b6953d96856d496: Status 404 returned error can't find the container with id e7971be82f38123c9e9aaceefc4c597df4dd898dd89ac4620b6953d96856d496 Nov 04 01:10:45.205830 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.205794 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:10:45.205931 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.205882 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:10:45.205931 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.205915 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:10:45.208107 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.208083 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2ee045-openshift-default-dockercfg-2pv54\"" Nov 04 01:10:45.230006 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.229975 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4"] Nov 04 01:10:45.279695 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.279670 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.279792 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.279728 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9kvm\" (UniqueName: \"kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-kube-api-access-z9kvm\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.279898 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.279875 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.279948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.279910 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.279948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.279940 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51822598-8499-472e-b731-4573487376e9-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.280020 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.279985 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51822598-8499-472e-b731-4573487376e9-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.280079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.280057 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.280195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.280178 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.280245 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.280215 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.381515 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.381474 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9kvm\" (UniqueName: \"kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-kube-api-access-z9kvm\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.381681 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.381542 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.381681 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.381578 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.381681 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.381607 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51822598-8499-472e-b731-4573487376e9-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.381681 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.381634 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51822598-8499-472e-b731-4573487376e9-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.381681 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.381668 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.381956 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.381744 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.381956 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.381777 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.381956 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.381832 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.382106 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.381991 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.382106 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.382039 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.382261 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.382237 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.382323 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.382270 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.382904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.382877 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51822598-8499-472e-b731-4573487376e9-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.384231 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.384188 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.384810 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.384786 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51822598-8499-472e-b731-4573487376e9-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.435729 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.435695 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.435873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.435756 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9kvm\" (UniqueName: \"kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-kube-api-access-z9kvm\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.465588 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.465550 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:45.674838 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.674743 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" event={"ID":"febf58ae-2630-40f9-a463-be0d0c583f2b","Type":"ContainerStarted","Data":"d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed"} Nov 04 01:10:45.674838 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.674811 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" event={"ID":"febf58ae-2630-40f9-a463-be0d0c583f2b","Type":"ContainerStarted","Data":"e7971be82f38123c9e9aaceefc4c597df4dd898dd89ac4620b6953d96856d496"} Nov 04 01:10:45.676464 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.676438 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" event={"ID":"a09a5559-a4e6-41f4-9dca-1806e1f61a8e","Type":"ContainerStarted","Data":"97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d"} Nov 04 01:10:45.774749 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.774716 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:45.790643 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.790471 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" podStartSLOduration=1.790455366 podStartE2EDuration="1.790455366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:45.790194631 +0000 UTC m=+791.406540397" watchObservedRunningTime="2026-11-04 01:10:45.790455366 +0000 UTC m=+791.406801131" Nov 04 01:10:45.955908 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.955793 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" podStartSLOduration=2.955775742 podStartE2EDuration="2.955775742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:45.908924066 +0000 UTC m=+791.525269830" watchObservedRunningTime="2026-11-04 01:10:45.955775742 +0000 UTC m=+791.572121507" Nov 04 01:10:45.956106 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.956085 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4"] Nov 04 01:10:45.963004 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:10:45.962972 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51822598_8499_472e_b731_4573487376e9.slice/crio-83c9b7330fffd2b8b22c779ff5c945c3cff769358112104717cb09b8065c70da WatchSource:0}: Error finding container 83c9b7330fffd2b8b22c779ff5c945c3cff769358112104717cb09b8065c70da: Status 404 returned error can't find the container with id 83c9b7330fffd2b8b22c779ff5c945c3cff769358112104717cb09b8065c70da Nov 04 01:10:45.965333 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.965299 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:10:45.965430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.965376 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:10:45.965430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:45.965422 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:10:46.108422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:46.108387 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:46.113488 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:46.113465 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:46.683398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:46.683363 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" event={"ID":"51822598-8499-472e-b731-4573487376e9","Type":"ContainerStarted","Data":"1489b1ed2688aa52fdd17db49523ad7248b665f48f81155aaaad20719b277eb9"} Nov 04 01:10:46.683659 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:46.683419 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" event={"ID":"51822598-8499-472e-b731-4573487376e9","Type":"ContainerStarted","Data":"83c9b7330fffd2b8b22c779ff5c945c3cff769358112104717cb09b8065c70da"} Nov 04 01:10:46.683836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:46.683812 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:46.684674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:46.684653 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:10:46.793655 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:46.793473 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:46.814896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:46.814681 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" podStartSLOduration=1.814658384 podStartE2EDuration="1.814658384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:46.805789212 +0000 UTC m=+792.422134976" watchObservedRunningTime="2026-11-04 01:10:46.814658384 +0000 UTC m=+792.431004148" Nov 04 01:10:47.466327 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:47.466289 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:47.471424 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:47.471396 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:47.689053 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:47.689020 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:47.689274 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:47.689078 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:47.689661 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:47.689639 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:10:47.689809 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:47.689795 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:10:53.257583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.257547 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8"] Nov 04 01:10:53.261372 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.261351 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" Nov 04 01:10:53.358815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.358779 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/028d9ee5-74e8-448d-b697-d636933c0643-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8\" (UID: \"028d9ee5-74e8-448d-b697-d636933c0643\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" Nov 04 01:10:53.358972 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.358829 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv5z5\" (UniqueName: \"kubernetes.io/projected/028d9ee5-74e8-448d-b697-d636933c0643-kube-api-access-gv5z5\") pod \"payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8\" (UID: \"028d9ee5-74e8-448d-b697-d636933c0643\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" Nov 04 01:10:53.367388 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.367362 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-e210ef\"" Nov 04 01:10:53.367514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.367486 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-e210ef-dockercfg-tz4s4\"" Nov 04 01:10:53.368963 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.368937 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8"] Nov 04 01:10:53.370177 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.370159 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z"] Nov 04 01:10:53.373553 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.373540 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" Nov 04 01:10:53.423801 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.423775 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z"] Nov 04 01:10:53.459571 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.459537 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv5z5\" (UniqueName: \"kubernetes.io/projected/028d9ee5-74e8-448d-b697-d636933c0643-kube-api-access-gv5z5\") pod \"payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8\" (UID: \"028d9ee5-74e8-448d-b697-d636933c0643\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" Nov 04 01:10:53.459728 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.459586 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xm9k\" (UniqueName: \"kubernetes.io/projected/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-kube-api-access-4xm9k\") pod \"payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z\" (UID: \"9840601d-5ec0-4a50-8088-64b0ac4fdc4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" Nov 04 01:10:53.459728 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.459608 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z\" (UID: \"9840601d-5ec0-4a50-8088-64b0ac4fdc4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" Nov 04 01:10:53.459828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.459803 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/028d9ee5-74e8-448d-b697-d636933c0643-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8\" (UID: \"028d9ee5-74e8-448d-b697-d636933c0643\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" Nov 04 01:10:53.460517 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.460500 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/028d9ee5-74e8-448d-b697-d636933c0643-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8\" (UID: \"028d9ee5-74e8-448d-b697-d636933c0643\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" Nov 04 01:10:53.514428 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.514353 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv5z5\" (UniqueName: \"kubernetes.io/projected/028d9ee5-74e8-448d-b697-d636933c0643-kube-api-access-gv5z5\") pod \"payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8\" (UID: \"028d9ee5-74e8-448d-b697-d636933c0643\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" Nov 04 01:10:53.561249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.561213 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xm9k\" (UniqueName: \"kubernetes.io/projected/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-kube-api-access-4xm9k\") pod \"payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z\" (UID: \"9840601d-5ec0-4a50-8088-64b0ac4fdc4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" Nov 04 01:10:53.561249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.561251 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z\" (UID: \"9840601d-5ec0-4a50-8088-64b0ac4fdc4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" Nov 04 01:10:53.561966 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.561946 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z\" (UID: \"9840601d-5ec0-4a50-8088-64b0ac4fdc4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" Nov 04 01:10:53.574060 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.574037 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" Nov 04 01:10:53.616775 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.616743 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xm9k\" (UniqueName: \"kubernetes.io/projected/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-kube-api-access-4xm9k\") pod \"payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z\" (UID: \"9840601d-5ec0-4a50-8088-64b0ac4fdc4e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" Nov 04 01:10:53.686666 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.686631 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" Nov 04 01:10:53.943257 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:53.943221 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8"] Nov 04 01:10:53.948504 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:10:53.948477 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028d9ee5_74e8_448d_b697_d636933c0643.slice/crio-8041366582beaeb823852f2c4660f73153d494e34948f2bb19073f306cfb3e65 WatchSource:0}: Error finding container 8041366582beaeb823852f2c4660f73153d494e34948f2bb19073f306cfb3e65: Status 404 returned error can't find the container with id 8041366582beaeb823852f2c4660f73153d494e34948f2bb19073f306cfb3e65 Nov 04 01:10:54.076577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.076541 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z"] Nov 04 01:10:54.082776 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:10:54.082744 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9840601d_5ec0_4a50_8088_64b0ac4fdc4e.slice/crio-59acb7c1da1a2c40992e9dc88dfb2ee9e56b3c9191bf78365d7c69fb1fe119a3 WatchSource:0}: Error finding container 59acb7c1da1a2c40992e9dc88dfb2ee9e56b3c9191bf78365d7c69fb1fe119a3: Status 404 returned error can't find the container with id 59acb7c1da1a2c40992e9dc88dfb2ee9e56b3c9191bf78365d7c69fb1fe119a3 Nov 04 01:10:54.390527 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.390490 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh"] Nov 04 01:10:54.395282 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.395256 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:54.448286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.448251 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-2fd27d-serving-cert\"" Nov 04 01:10:54.450272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.450243 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh"] Nov 04 01:10:54.472522 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.472489 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljqwl\" (UniqueName: \"kubernetes.io/projected/a5958d8a-1f07-45d1-8410-57494d457efe-kube-api-access-ljqwl\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:54.472698 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.472534 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5958d8a-1f07-45d1-8410-57494d457efe-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:54.472698 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.472556 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:54.472698 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.472669 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-metrics-certs\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:54.574080 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.574044 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljqwl\" (UniqueName: \"kubernetes.io/projected/a5958d8a-1f07-45d1-8410-57494d457efe-kube-api-access-ljqwl\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:54.574301 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.574102 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5958d8a-1f07-45d1-8410-57494d457efe-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:54.574301 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.574255 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:54.574437 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.574320 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-metrics-certs\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:54.574437 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:10:54.574375 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-serving-cert: secret "maas-api-e2e-worker-w2-2fd27d-serving-cert" not found Nov 04 01:10:54.574527 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:10:54.574456 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-maas-api-tls podName:a5958d8a-1f07-45d1-8410-57494d457efe nodeName:}" failed. No retries permitted until 2026-11-04 01:10:55.074439079 +0000 UTC m=+800.690784833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-maas-api-tls") pod "maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" (UID: "a5958d8a-1f07-45d1-8410-57494d457efe") : secret "maas-api-e2e-worker-w2-2fd27d-serving-cert" not found Nov 04 01:10:54.574892 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.574871 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5958d8a-1f07-45d1-8410-57494d457efe-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:54.576824 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.576800 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-metrics-certs\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:54.631153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.631082 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljqwl\" (UniqueName: \"kubernetes.io/projected/a5958d8a-1f07-45d1-8410-57494d457efe-kube-api-access-ljqwl\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:54.723203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.723095 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" event={"ID":"9840601d-5ec0-4a50-8088-64b0ac4fdc4e","Type":"ContainerStarted","Data":"59acb7c1da1a2c40992e9dc88dfb2ee9e56b3c9191bf78365d7c69fb1fe119a3"} Nov 04 01:10:54.724520 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:54.724494 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" event={"ID":"028d9ee5-74e8-448d-b697-d636933c0643","Type":"ContainerStarted","Data":"8041366582beaeb823852f2c4660f73153d494e34948f2bb19073f306cfb3e65"} Nov 04 01:10:55.080496 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:55.080406 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:55.082929 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:55.082900 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:55.310856 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:55.310782 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:55.700288 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:55.700204 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh"] Nov 04 01:10:55.707791 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:10:55.707641 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5958d8a_1f07_45d1_8410_57494d457efe.slice/crio-7d61296facaf915ba4a12fbfc58b58efdc8582bd018340f5f941f6915cfdf572 WatchSource:0}: Error finding container 7d61296facaf915ba4a12fbfc58b58efdc8582bd018340f5f941f6915cfdf572: Status 404 returned error can't find the container with id 7d61296facaf915ba4a12fbfc58b58efdc8582bd018340f5f941f6915cfdf572 Nov 04 01:10:55.732057 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:55.732018 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" event={"ID":"a5958d8a-1f07-45d1-8410-57494d457efe","Type":"ContainerStarted","Data":"7d61296facaf915ba4a12fbfc58b58efdc8582bd018340f5f941f6915cfdf572"} Nov 04 01:10:57.751566 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:57.751513 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" event={"ID":"9840601d-5ec0-4a50-8088-64b0ac4fdc4e","Type":"ContainerStarted","Data":"13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e"} Nov 04 01:10:57.752040 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:57.751585 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" Nov 04 01:10:57.753182 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:57.753153 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" event={"ID":"028d9ee5-74e8-448d-b697-d636933c0643","Type":"ContainerStarted","Data":"0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b"} Nov 04 01:10:57.753321 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:57.753281 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" Nov 04 01:10:57.754578 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:57.754556 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" event={"ID":"a5958d8a-1f07-45d1-8410-57494d457efe","Type":"ContainerStarted","Data":"f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af"} Nov 04 01:10:57.754698 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:57.754675 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:10:57.861144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:57.860461 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" podStartSLOduration=1.833045815 podStartE2EDuration="4.860444183s" totalImagesPullingTime="3.027398368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:57.859841801 +0000 UTC m=+803.476187565" watchObservedRunningTime="2026-11-04 01:10:57.860444183 +0000 UTC m=+803.476789947" Nov 04 01:10:57.970564 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:57.970505 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" podStartSLOduration=1.8408604579999999 podStartE2EDuration="4.970490725s" totalImagesPullingTime="3.129630267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:57.969355635 +0000 UTC m=+803.585701397" watchObservedRunningTime="2026-11-04 01:10:57.970490725 +0000 UTC m=+803.586836528" Nov 04 01:10:58.080820 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:10:58.080716 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" podStartSLOduration=2.995858208 podStartE2EDuration="4.080700069s" totalImagesPullingTime="1.084841861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:58.07910448 +0000 UTC m=+803.695450243" watchObservedRunningTime="2026-11-04 01:10:58.080700069 +0000 UTC m=+803.697045831" Nov 04 01:11:03.261144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:03.256769 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:11:03.766249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:03.766208 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:11:08.763874 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:08.763847 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" Nov 04 01:11:08.764302 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:08.764004 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" Nov 04 01:11:10.649602 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.649565 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888"] Nov 04 01:11:10.654408 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.654389 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" Nov 04 01:11:10.667266 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.667234 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888\" (UID: \"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" Nov 04 01:11:10.667412 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.667393 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28tq8\" (UniqueName: \"kubernetes.io/projected/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-kube-api-access-28tq8\") pod \"payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888\" (UID: \"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" Nov 04 01:11:10.756079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.755330 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c563ee\"" Nov 04 01:11:10.767343 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.767314 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c563ee-dockercfg-v8zst\"" Nov 04 01:11:10.768147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.768087 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28tq8\" (UniqueName: \"kubernetes.io/projected/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-kube-api-access-28tq8\") pod \"payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888\" (UID: \"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" Nov 04 01:11:10.768288 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.768166 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888\" (UID: \"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" Nov 04 01:11:10.769303 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.769109 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888\" (UID: \"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" Nov 04 01:11:10.778042 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.778011 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888"] Nov 04 01:11:10.791503 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.791470 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg"] Nov 04 01:11:10.797754 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.797716 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" Nov 04 01:11:10.833002 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.832971 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tq8\" (UniqueName: \"kubernetes.io/projected/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-kube-api-access-28tq8\") pod \"payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888\" (UID: \"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" Nov 04 01:11:10.849071 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.849040 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg"] Nov 04 01:11:10.868739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.868699 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e57d7b-8a83-465f-a030-d98122b1935f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg\" (UID: \"b4e57d7b-8a83-465f-a030-d98122b1935f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" Nov 04 01:11:10.868898 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.868873 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjwjm\" (UniqueName: \"kubernetes.io/projected/b4e57d7b-8a83-465f-a030-d98122b1935f-kube-api-access-qjwjm\") pod \"payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg\" (UID: \"b4e57d7b-8a83-465f-a030-d98122b1935f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" Nov 04 01:11:10.970323 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.970231 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e57d7b-8a83-465f-a030-d98122b1935f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg\" (UID: \"b4e57d7b-8a83-465f-a030-d98122b1935f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" Nov 04 01:11:10.970503 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.970355 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjwjm\" (UniqueName: \"kubernetes.io/projected/b4e57d7b-8a83-465f-a030-d98122b1935f-kube-api-access-qjwjm\") pod \"payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg\" (UID: \"b4e57d7b-8a83-465f-a030-d98122b1935f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" Nov 04 01:11:10.970941 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:10.970920 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e57d7b-8a83-465f-a030-d98122b1935f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg\" (UID: \"b4e57d7b-8a83-465f-a030-d98122b1935f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" Nov 04 01:11:11.004860 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:11.004818 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" Nov 04 01:11:11.025146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:11.025103 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjwjm\" (UniqueName: \"kubernetes.io/projected/b4e57d7b-8a83-465f-a030-d98122b1935f-kube-api-access-qjwjm\") pod \"payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg\" (UID: \"b4e57d7b-8a83-465f-a030-d98122b1935f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" Nov 04 01:11:11.111999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:11.111958 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" Nov 04 01:11:11.383352 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:11.383321 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888"] Nov 04 01:11:11.388636 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:11:11.388610 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e7cd47_ec58_4d25_abac_8b3e3001f3a0.slice/crio-0575b80ecebbdca1a2a905f85a7ac6fd823c06725e81f383c273c4a166fc828f WatchSource:0}: Error finding container 0575b80ecebbdca1a2a905f85a7ac6fd823c06725e81f383c273c4a166fc828f: Status 404 returned error can't find the container with id 0575b80ecebbdca1a2a905f85a7ac6fd823c06725e81f383c273c4a166fc828f Nov 04 01:11:11.494715 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:11.494680 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg"] Nov 04 01:11:11.500536 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:11:11.500504 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e57d7b_8a83_465f_a030_d98122b1935f.slice/crio-5108d3e486184fa9abfc9f228b6e87b2d34ae9df3bacacb8d7da993dab98c434 WatchSource:0}: Error finding container 5108d3e486184fa9abfc9f228b6e87b2d34ae9df3bacacb8d7da993dab98c434: Status 404 returned error can't find the container with id 5108d3e486184fa9abfc9f228b6e87b2d34ae9df3bacacb8d7da993dab98c434 Nov 04 01:11:11.834768 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:11.834677 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" event={"ID":"b4e57d7b-8a83-465f-a030-d98122b1935f","Type":"ContainerStarted","Data":"1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b"} Nov 04 01:11:11.834768 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:11.834727 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" event={"ID":"b4e57d7b-8a83-465f-a030-d98122b1935f","Type":"ContainerStarted","Data":"5108d3e486184fa9abfc9f228b6e87b2d34ae9df3bacacb8d7da993dab98c434"} Nov 04 01:11:11.834768 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:11.834756 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" Nov 04 01:11:11.836244 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:11.836219 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" event={"ID":"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0","Type":"ContainerStarted","Data":"ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025"} Nov 04 01:11:11.836349 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:11.836260 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" event={"ID":"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0","Type":"ContainerStarted","Data":"0575b80ecebbdca1a2a905f85a7ac6fd823c06725e81f383c273c4a166fc828f"} Nov 04 01:11:11.836394 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:11.836344 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" Nov 04 01:11:11.945529 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:11.945473 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" podStartSLOduration=1.945457206 podStartE2EDuration="1.945457206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:11.944004845 +0000 UTC m=+817.560350611" watchObservedRunningTime="2026-11-04 01:11:11.945457206 +0000 UTC m=+817.561802965" Nov 04 01:11:12.055231 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:12.055167 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" podStartSLOduration=2.055146326 podStartE2EDuration="2.055146326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:12.053341156 +0000 UTC m=+817.669686920" watchObservedRunningTime="2026-11-04 01:11:12.055146326 +0000 UTC m=+817.671492091" Nov 04 01:11:13.055559 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.055523 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:11:13.385610 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.385581 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv"] Nov 04 01:11:13.389869 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.389845 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" Nov 04 01:11:13.443920 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.443888 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv"] Nov 04 01:11:13.489661 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.489631 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e65b23\"" Nov 04 01:11:13.489852 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.489696 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-e65b23-dockercfg-4vcsd\"" Nov 04 01:11:13.495089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.495061 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxwz\" (UniqueName: \"kubernetes.io/projected/41127884-4d48-4b30-a227-2a34a270c863-kube-api-access-smxwz\") pod \"payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv\" (UID: \"41127884-4d48-4b30-a227-2a34a270c863\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" Nov 04 01:11:13.495181 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.495103 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41127884-4d48-4b30-a227-2a34a270c863-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv\" (UID: \"41127884-4d48-4b30-a227-2a34a270c863\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" Nov 04 01:11:13.596785 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.596740 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxwz\" (UniqueName: \"kubernetes.io/projected/41127884-4d48-4b30-a227-2a34a270c863-kube-api-access-smxwz\") pod \"payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv\" (UID: \"41127884-4d48-4b30-a227-2a34a270c863\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" Nov 04 01:11:13.597081 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.597049 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41127884-4d48-4b30-a227-2a34a270c863-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv\" (UID: \"41127884-4d48-4b30-a227-2a34a270c863\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" Nov 04 01:11:13.597942 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.597919 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41127884-4d48-4b30-a227-2a34a270c863-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv\" (UID: \"41127884-4d48-4b30-a227-2a34a270c863\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" Nov 04 01:11:13.651751 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.651678 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxwz\" (UniqueName: \"kubernetes.io/projected/41127884-4d48-4b30-a227-2a34a270c863-kube-api-access-smxwz\") pod \"payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv\" (UID: \"41127884-4d48-4b30-a227-2a34a270c863\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" Nov 04 01:11:13.703329 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:13.703294 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" Nov 04 01:11:14.088349 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:14.076072 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv"] Nov 04 01:11:14.126666 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:11:14.126627 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41127884_4d48_4b30_a227_2a34a270c863.slice/crio-d98996bf48846555919d9ab02daca0ec1c8f68a1388b0cd425931da45a0f53c2 WatchSource:0}: Error finding container d98996bf48846555919d9ab02daca0ec1c8f68a1388b0cd425931da45a0f53c2: Status 404 returned error can't find the container with id d98996bf48846555919d9ab02daca0ec1c8f68a1388b0cd425931da45a0f53c2 Nov 04 01:11:14.852049 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:14.852005 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" event={"ID":"41127884-4d48-4b30-a227-2a34a270c863","Type":"ContainerStarted","Data":"3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c"} Nov 04 01:11:14.852242 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:14.852071 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" event={"ID":"41127884-4d48-4b30-a227-2a34a270c863","Type":"ContainerStarted","Data":"d98996bf48846555919d9ab02daca0ec1c8f68a1388b0cd425931da45a0f53c2"} Nov 04 01:11:14.852242 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:14.852184 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" Nov 04 01:11:14.969144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:14.966352 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" podStartSLOduration=1.966325987 podStartE2EDuration="1.966325987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:14.965538694 +0000 UTC m=+820.581884458" watchObservedRunningTime="2026-11-04 01:11:14.966325987 +0000 UTC m=+820.582671751" Nov 04 01:11:15.127623 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:11:15.127542 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:11:15.533195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.528032 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq"] Nov 04 01:11:15.552485 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.542447 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:15.618140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.596417 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq"] Nov 04 01:11:15.640486 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.640437 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbv5q\" (UniqueName: \"kubernetes.io/projected/b8bf9c7a-9890-4a39-98eb-1a2044daf167-kube-api-access-lbv5q\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:15.640660 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.640618 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8bf9c7a-9890-4a39-98eb-1a2044daf167-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:15.640731 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.640712 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:15.640793 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.640777 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-metrics-certs\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:15.641316 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.641295 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-e210ef-serving-cert\"" Nov 04 01:11:15.746622 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.746570 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8bf9c7a-9890-4a39-98eb-1a2044daf167-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:15.746819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.746638 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:15.746819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.746666 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-metrics-certs\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:15.746819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.746694 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbv5q\" (UniqueName: \"kubernetes.io/projected/b8bf9c7a-9890-4a39-98eb-1a2044daf167-kube-api-access-lbv5q\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:15.747177 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:11:15.747152 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-serving-cert: secret "maas-api-e2e-worker-w5-e210ef-serving-cert" not found Nov 04 01:11:15.747262 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:11:15.747228 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-maas-api-tls podName:b8bf9c7a-9890-4a39-98eb-1a2044daf167 nodeName:}" failed. No retries permitted until 2026-11-04 01:11:16.247206623 +0000 UTC m=+821.863552368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-maas-api-tls") pod "maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" (UID: "b8bf9c7a-9890-4a39-98eb-1a2044daf167") : secret "maas-api-e2e-worker-w5-e210ef-serving-cert" not found Nov 04 01:11:15.747829 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.747761 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8bf9c7a-9890-4a39-98eb-1a2044daf167-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:15.750806 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.750455 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-metrics-certs\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:15.807057 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:15.806980 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbv5q\" (UniqueName: \"kubernetes.io/projected/b8bf9c7a-9890-4a39-98eb-1a2044daf167-kube-api-access-lbv5q\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:16.253597 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:16.253561 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:16.257148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:16.257106 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:16.469913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:16.469322 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:16.911216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:16.911175 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq"] Nov 04 01:11:16.917687 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:11:16.917029 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8bf9c7a_9890_4a39_98eb_1a2044daf167.slice/crio-b92432604c00fa24c224ca3194a49188dd19cdb84c894a23ed073105aba3b876 WatchSource:0}: Error finding container b92432604c00fa24c224ca3194a49188dd19cdb84c894a23ed073105aba3b876: Status 404 returned error can't find the container with id b92432604c00fa24c224ca3194a49188dd19cdb84c894a23ed073105aba3b876 Nov 04 01:11:17.915607 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:17.915567 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" event={"ID":"b8bf9c7a-9890-4a39-98eb-1a2044daf167","Type":"ContainerStarted","Data":"b92432604c00fa24c224ca3194a49188dd19cdb84c894a23ed073105aba3b876"} Nov 04 01:11:18.923942 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:18.923906 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" event={"ID":"b8bf9c7a-9890-4a39-98eb-1a2044daf167","Type":"ContainerStarted","Data":"2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea"} Nov 04 01:11:18.924427 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:18.923990 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:19.042982 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:19.042932 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" podStartSLOduration=3.030140802 podStartE2EDuration="4.042918297s" totalImagesPullingTime="1.012777495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:19.041559342 +0000 UTC m=+824.657905107" watchObservedRunningTime="2026-11-04 01:11:19.042918297 +0000 UTC m=+824.659264059" Nov 04 01:11:22.843889 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.843800 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" Nov 04 01:11:22.844331 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.844073 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" Nov 04 01:11:22.947079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.947046 5786 generic.go:356] "Generic (PLEG): container finished" podID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerID="97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d" exitCode=0 Nov 04 01:11:22.947254 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.947128 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" event={"ID":"a09a5559-a4e6-41f4-9dca-1806e1f61a8e","Type":"ContainerDied","Data":"97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d"} Nov 04 01:11:22.947558 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.947539 5786 scope.go:122] "RemoveContainer" containerID="97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d" Nov 04 01:11:22.948875 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.948856 5786 generic.go:356] "Generic (PLEG): container finished" podID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerID="d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed" exitCode=0 Nov 04 01:11:22.948958 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.948897 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" event={"ID":"febf58ae-2630-40f9-a463-be0d0c583f2b","Type":"ContainerDied","Data":"d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed"} Nov 04 01:11:22.949010 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.948990 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:11:22.949067 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.949053 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:11:22.949103 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.949090 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:11:22.949199 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.949186 5786 scope.go:122] "RemoveContainer" containerID="d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed" Nov 04 01:11:22.950281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.950254 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:11:22.950385 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.950309 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:11:22.950385 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:22.950347 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:11:23.955975 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:23.955931 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" event={"ID":"febf58ae-2630-40f9-a463-be0d0c583f2b","Type":"ContainerStarted","Data":"b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79"} Nov 04 01:11:23.957882 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:23.957862 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" event={"ID":"a09a5559-a4e6-41f4-9dca-1806e1f61a8e","Type":"ContainerStarted","Data":"ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832"} Nov 04 01:11:24.108553 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:24.108515 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:11:24.108917 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:24.108893 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:24.109066 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:24.109043 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 01:11:24.127240 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:24.127210 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:11:24.777146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:24.775284 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:11:24.777146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:24.775723 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 04 01:11:24.777146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:24.775877 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 04 01:11:24.937043 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:24.937006 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:11:25.136193 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:25.134483 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:25.136193 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:25.134549 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 01:11:25.777537 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:25.777498 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 04 01:11:25.777815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:25.777785 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 04 01:11:25.887333 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:25.886315 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" Nov 04 01:11:26.111144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:26.108445 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:26.111144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:26.108505 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 01:11:26.775630 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:26.775588 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 04 01:11:26.776284 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:26.776248 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 04 01:11:27.111537 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:27.111494 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:27.111702 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:27.111563 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 01:11:27.780563 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:27.780521 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 04 01:11:27.781213 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:27.781176 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 04 01:11:28.109154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:28.108541 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:28.109154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:28.108608 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 01:11:28.778461 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:28.778422 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 04 01:11:28.778748 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:28.778712 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 04 01:11:29.111502 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:29.111471 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 01:11:29.111971 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:29.111938 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 01:11:29.779261 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:29.779230 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:11:29.779659 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:29.779636 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:11:29.780327 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:29.780308 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:11:30.112356 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:30.112326 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:11:30.112817 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:30.112740 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:11:30.113408 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:30.113390 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:11:35.250566 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:35.250535 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:11:41.241369 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.241334 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb"] Nov 04 01:11:41.248207 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.248181 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.299801 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.299766 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb"] Nov 04 01:11:41.345278 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.345246 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-2ee045-serving-cert\"" Nov 04 01:11:41.404311 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.404274 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2ww7\" (UniqueName: \"kubernetes.io/projected/72c067af-133f-494e-b755-926e1047f874-kube-api-access-k2ww7\") pod \"maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.404479 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.404384 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c067af-133f-494e-b755-926e1047f874-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.404575 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.404549 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.404623 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.404607 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-metrics-certs\") pod \"maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.506259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.506166 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.506259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.506228 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-metrics-certs\") pod \"maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.506490 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.506296 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2ww7\" (UniqueName: \"kubernetes.io/projected/72c067af-133f-494e-b755-926e1047f874-kube-api-access-k2ww7\") pod \"maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.506490 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.506347 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c067af-133f-494e-b755-926e1047f874-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.506974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.506945 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c067af-133f-494e-b755-926e1047f874-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.508944 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.508918 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-metrics-certs\") pod \"maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.509064 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.509000 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.560701 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.560661 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2ww7\" (UniqueName: \"kubernetes.io/projected/72c067af-133f-494e-b755-926e1047f874-kube-api-access-k2ww7\") pod \"maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:41.860823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:41.860789 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:42.234073 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:42.233993 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb"] Nov 04 01:11:42.239740 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:11:42.239710 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c067af_133f_494e_b755_926e1047f874.slice/crio-4839482bc2c64cab34b6925ddd69b0e9a569dc3c6a886dea5c033c66312a9070 WatchSource:0}: Error finding container 4839482bc2c64cab34b6925ddd69b0e9a569dc3c6a886dea5c033c66312a9070: Status 404 returned error can't find the container with id 4839482bc2c64cab34b6925ddd69b0e9a569dc3c6a886dea5c033c66312a9070 Nov 04 01:11:43.111785 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:43.111745 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" event={"ID":"72c067af-133f-494e-b755-926e1047f874","Type":"ContainerStarted","Data":"4839482bc2c64cab34b6925ddd69b0e9a569dc3c6a886dea5c033c66312a9070"} Nov 04 01:11:44.117689 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:44.117657 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" event={"ID":"72c067af-133f-494e-b755-926e1047f874","Type":"ContainerStarted","Data":"0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1"} Nov 04 01:11:44.118107 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:44.117842 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:44.228474 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:44.228416 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" podStartSLOduration=2.343929078 podStartE2EDuration="3.228400339s" totalImagesPullingTime="884.471261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:44.226559058 +0000 UTC m=+849.842904821" watchObservedRunningTime="2026-11-04 01:11:44.228400339 +0000 UTC m=+849.844746101" Nov 04 01:11:45.087277 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:11:45.087237 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:11:47.693922 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:47.693890 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:11:50.128274 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:50.128245 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:11:57.215919 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:57.215885 5786 generic.go:356] "Generic (PLEG): container finished" podID="51822598-8499-472e-b731-4573487376e9" containerID="1489b1ed2688aa52fdd17db49523ad7248b665f48f81155aaaad20719b277eb9" exitCode=0 Nov 04 01:11:57.216357 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:57.215928 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" event={"ID":"51822598-8499-472e-b731-4573487376e9","Type":"ContainerDied","Data":"1489b1ed2688aa52fdd17db49523ad7248b665f48f81155aaaad20719b277eb9"} Nov 04 01:11:57.216357 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:57.216335 5786 scope.go:122] "RemoveContainer" containerID="1489b1ed2688aa52fdd17db49523ad7248b665f48f81155aaaad20719b277eb9" Nov 04 01:11:57.217655 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:57.217624 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:11:57.217720 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:57.217694 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:11:57.217756 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:57.217725 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:11:58.225826 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:58.225783 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" event={"ID":"51822598-8499-472e-b731-4573487376e9","Type":"ContainerStarted","Data":"01865512c7354a34e7883e11e1c85f0c16a34c3eab7b1aabcfca81d6d458eed6"} Nov 04 01:11:58.471515 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:58.471480 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:11:58.478347 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:58.478246 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:11:58.478620 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:58.478587 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 04 01:11:59.472144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:59.469505 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:11:59.472144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:11:59.469572 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 04 01:12:00.470154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:00.467531 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:12:00.470154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:00.467598 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 04 01:12:01.471564 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:01.471524 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:12:01.472227 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:01.472192 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 04 01:12:02.481143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:02.480502 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 04 01:12:02.481143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:02.480570 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 04 01:12:02.689048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:02.689006 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:12:03.531705 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:03.531668 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:12:03.533399 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:03.533374 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:12:09.562875 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:09.562837 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:12:15.086742 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:12:15.086701 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:12:31.344974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:31.344936 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:12:32.389430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.389393 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj"] Nov 04 01:12:32.393567 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.393544 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.493276 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.493235 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2fd27d\"/\"premium-w2-2fd27d-kserve-self-signed-certs\"" Nov 04 01:12:32.493276 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.493251 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2fd27d\"/\"openshift-service-ca.crt\"" Nov 04 01:12:32.493276 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.493240 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2fd27d\"/\"default-dockercfg-kvnqv\"" Nov 04 01:12:32.493509 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.493385 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2fd27d\"/\"kube-root-ca.crt\"" Nov 04 01:12:32.494878 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.494857 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj"] Nov 04 01:12:32.556990 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.556944 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-home\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.557284 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.557263 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-dshm\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.557470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.557450 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mczzd\" (UniqueName: \"kubernetes.io/projected/e54f93ab-904a-4feb-a157-fa470223d756-kube-api-access-mczzd\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.557641 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.557624 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-model-cache\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.557783 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.557767 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-tmp-dir\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.559846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.559811 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f93ab-904a-4feb-a157-fa470223d756-tls-certs\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.660495 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.660401 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mczzd\" (UniqueName: \"kubernetes.io/projected/e54f93ab-904a-4feb-a157-fa470223d756-kube-api-access-mczzd\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.660495 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.660488 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-model-cache\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.660746 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.660534 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-tmp-dir\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.660746 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.660573 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f93ab-904a-4feb-a157-fa470223d756-tls-certs\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.660746 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.660660 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-home\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.660746 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.660715 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-dshm\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.660993 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.660970 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-model-cache\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.661067 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.661040 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-home\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.661224 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.661205 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-tmp-dir\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.663012 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.662991 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-dshm\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.663164 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.663146 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f93ab-904a-4feb-a157-fa470223d756-tls-certs\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:32.715475 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:32.715445 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mczzd\" (UniqueName: \"kubernetes.io/projected/e54f93ab-904a-4feb-a157-fa470223d756-kube-api-access-mczzd\") pod \"premium-w2-2fd27d-kserve-66df56bcf-b4vgj\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:33.006422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:33.006335 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:33.378872 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:33.378834 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj"] Nov 04 01:12:33.383763 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:12:33.383735 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode54f93ab_904a_4feb_a157_fa470223d756.slice/crio-6c0704066a3036d080025a161227e2f59b4387422b28c5b6105260b1e17548d1 WatchSource:0}: Error finding container 6c0704066a3036d080025a161227e2f59b4387422b28c5b6105260b1e17548d1: Status 404 returned error can't find the container with id 6c0704066a3036d080025a161227e2f59b4387422b28c5b6105260b1e17548d1 Nov 04 01:12:33.406651 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:33.406619 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" event={"ID":"e54f93ab-904a-4feb-a157-fa470223d756","Type":"ContainerStarted","Data":"6c0704066a3036d080025a161227e2f59b4387422b28c5b6105260b1e17548d1"} Nov 04 01:12:34.414506 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:34.414470 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" event={"ID":"e54f93ab-904a-4feb-a157-fa470223d756","Type":"ContainerStarted","Data":"d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11"} Nov 04 01:12:34.527692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:34.527642 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podStartSLOduration=2.5276284540000002 podStartE2EDuration="2.527628454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:34.52732403 +0000 UTC m=+900.143669793" watchObservedRunningTime="2026-11-04 01:12:34.527628454 +0000 UTC m=+900.143974217" Nov 04 01:12:34.579798 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:34.579765 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:12:34.581400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:34.581377 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:12:35.006468 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:35.006432 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:43.007378 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:43.007336 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:43.019971 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:43.019943 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:43.486888 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:43.486859 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:12:44.595374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.595290 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk"] Nov 04 01:12:44.599350 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.599328 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.654822 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.654785 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk"] Nov 04 01:12:44.699426 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.699390 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c563ee\"/\"openshift-service-ca.crt\"" Nov 04 01:12:44.699606 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.699588 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c563ee\"/\"kube-root-ca.crt\"" Nov 04 01:12:44.699674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.699636 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c563ee\"/\"simulator-w1-c563ee-kserve-self-signed-certs\"" Nov 04 01:12:44.699713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.699671 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c563ee\"/\"default-dockercfg-f4xf6\"" Nov 04 01:12:44.707179 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.707160 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-dshm\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.707248 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.707197 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-home\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.707248 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.707225 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e80b3fd-636d-4b64-912a-57b1e782dab7-tls-certs\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.707364 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.707348 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-model-cache\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.707413 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.707380 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-tmp-dir\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.707479 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.707465 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w429n\" (UniqueName: \"kubernetes.io/projected/5e80b3fd-636d-4b64-912a-57b1e782dab7-kube-api-access-w429n\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.808920 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.808880 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w429n\" (UniqueName: \"kubernetes.io/projected/5e80b3fd-636d-4b64-912a-57b1e782dab7-kube-api-access-w429n\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.809079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.809023 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-dshm\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.809079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.809051 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-home\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.809079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.809078 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e80b3fd-636d-4b64-912a-57b1e782dab7-tls-certs\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.809306 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.809110 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-model-cache\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.809306 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.809166 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-tmp-dir\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.809565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.809543 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-home\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.809642 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.809594 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-model-cache\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.809642 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.809611 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-tmp-dir\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.811398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.811377 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-dshm\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.811578 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.811558 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e80b3fd-636d-4b64-912a-57b1e782dab7-tls-certs\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.863978 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.863904 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w429n\" (UniqueName: \"kubernetes.io/projected/5e80b3fd-636d-4b64-912a-57b1e782dab7-kube-api-access-w429n\") pod \"simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:44.911186 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:44.911152 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:45.112985 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:12:45.112948 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:12:45.293914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:45.293829 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk"] Nov 04 01:12:45.299452 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:12:45.299423 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e80b3fd_636d_4b64_912a_57b1e782dab7.slice/crio-f474f6887514690c499a6af407c9bef7290c77b446145b78b274755e12d40d13 WatchSource:0}: Error finding container f474f6887514690c499a6af407c9bef7290c77b446145b78b274755e12d40d13: Status 404 returned error can't find the container with id f474f6887514690c499a6af407c9bef7290c77b446145b78b274755e12d40d13 Nov 04 01:12:45.487686 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:45.487645 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" event={"ID":"5e80b3fd-636d-4b64-912a-57b1e782dab7","Type":"ContainerStarted","Data":"83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec"} Nov 04 01:12:45.487859 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:45.487698 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" event={"ID":"5e80b3fd-636d-4b64-912a-57b1e782dab7","Type":"ContainerStarted","Data":"f474f6887514690c499a6af407c9bef7290c77b446145b78b274755e12d40d13"} Nov 04 01:12:45.597759 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:45.597705 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podStartSLOduration=1.59769297 podStartE2EDuration="1.59769297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:12:45.597107971 +0000 UTC m=+911.213453731" watchObservedRunningTime="2026-11-04 01:12:45.59769297 +0000 UTC m=+911.214038734" Nov 04 01:12:45.911464 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:45.911374 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:54.716466 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:54.716426 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:12:54.911676 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:54.911634 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:54.935531 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:54.935487 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:12:55.548061 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:12:55.548036 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:13:13.532471 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.532435 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z"] Nov 04 01:13:13.536410 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.536388 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.609325 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.609290 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-home\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.609480 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.609337 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tmp-dir\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.609480 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.609423 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-model-cache\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.609480 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.609464 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tls-certs\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.609584 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.609496 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b72p\" (UniqueName: \"kubernetes.io/projected/f457f48b-d5aa-43d0-ab4d-0df141a949c1-kube-api-access-7b72p\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.609625 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.609613 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-dshm\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.636778 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.636743 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2ee045\"/\"kube-root-ca.crt\"" Nov 04 01:13:13.636932 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.636809 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2ee045\"/\"openshift-service-ca.crt\"" Nov 04 01:13:13.636932 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.636821 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2ee045\"/\"default-dockercfg-pjk8b\"" Nov 04 01:13:13.636932 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.636918 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2ee045\"/\"simulator-w3-2ee045-kserve-self-signed-certs\"" Nov 04 01:13:13.638443 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.638423 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z"] Nov 04 01:13:13.712781 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.712746 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tls-certs\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.714679 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.714641 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b72p\" (UniqueName: \"kubernetes.io/projected/f457f48b-d5aa-43d0-ab4d-0df141a949c1-kube-api-access-7b72p\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.714847 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.714740 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-dshm\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.714916 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.714884 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-home\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.714972 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.714929 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tmp-dir\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.714972 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.714960 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-model-cache\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.715388 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.715370 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-model-cache\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.716287 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.716215 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-home\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.716401 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.716315 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tmp-dir\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.718374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.718341 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-dshm\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.718458 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.718384 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tls-certs\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.774646 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.774614 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b72p\" (UniqueName: \"kubernetes.io/projected/f457f48b-d5aa-43d0-ab4d-0df141a949c1-kube-api-access-7b72p\") pod \"simulator-w3-2ee045-kserve-bf9999b79-hjb8z\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:13.850138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:13.850087 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:14.219781 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:14.219696 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z"] Nov 04 01:13:14.225606 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:13:14.225573 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf457f48b_d5aa_43d0_ab4d_0df141a949c1.slice/crio-0eb9bc1e946e76feb9838690fed52e3a75e17abb3507f6802fd0216bcdc4aac1 WatchSource:0}: Error finding container 0eb9bc1e946e76feb9838690fed52e3a75e17abb3507f6802fd0216bcdc4aac1: Status 404 returned error can't find the container with id 0eb9bc1e946e76feb9838690fed52e3a75e17abb3507f6802fd0216bcdc4aac1 Nov 04 01:13:14.633541 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:14.633502 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" event={"ID":"f457f48b-d5aa-43d0-ab4d-0df141a949c1","Type":"ContainerStarted","Data":"3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307"} Nov 04 01:13:14.633932 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:14.633551 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" event={"ID":"f457f48b-d5aa-43d0-ab4d-0df141a949c1","Type":"ContainerStarted","Data":"0eb9bc1e946e76feb9838690fed52e3a75e17abb3507f6802fd0216bcdc4aac1"} Nov 04 01:13:14.742705 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:14.742659 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podStartSLOduration=1.7426452989999999 podStartE2EDuration="1.742645299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:14.742447429 +0000 UTC m=+940.358793195" watchObservedRunningTime="2026-11-04 01:13:14.742645299 +0000 UTC m=+940.358991105" Nov 04 01:13:14.850926 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:14.850889 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:15.086909 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:13:15.086827 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:13:18.166419 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:18.166384 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:13:23.850744 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:23.850705 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:23.863192 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:23.863168 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:24.702234 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:24.702206 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:13:45.088034 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:13:45.087993 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:13:53.857703 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:53.857664 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:13:55.635737 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.635694 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck"] Nov 04 01:13:55.640082 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.640060 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.659193 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.659165 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-home\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.659316 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.659217 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-tmp-dir\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.659365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.659339 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7102928-5338-4f63-950b-0900b80b07d3-tls-certs\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.659412 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.659390 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-model-cache\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.659504 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.659489 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-dshm\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.659564 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.659533 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qztnq\" (UniqueName: \"kubernetes.io/projected/c7102928-5338-4f63-950b-0900b80b07d3-kube-api-access-qztnq\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.690551 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.690519 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e210ef\"/\"kube-root-ca.crt\"" Nov 04 01:13:55.738139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.738086 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e210ef\"/\"openshift-service-ca.crt\"" Nov 04 01:13:55.738139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.738091 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e210ef\"/\"default-dockercfg-mblzv\"" Nov 04 01:13:55.738403 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.738151 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e210ef\"/\"premium-w5-e210ef-kserve-self-signed-certs\"" Nov 04 01:13:55.740246 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.740225 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck"] Nov 04 01:13:55.760758 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.760727 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-home\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.760904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.760772 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-tmp-dir\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.760904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.760843 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7102928-5338-4f63-950b-0900b80b07d3-tls-certs\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.760904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.760897 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-model-cache\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.761019 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.760944 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-dshm\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.761019 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.760985 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qztnq\" (UniqueName: \"kubernetes.io/projected/c7102928-5338-4f63-950b-0900b80b07d3-kube-api-access-qztnq\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.761264 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.761242 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-home\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.761354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.761333 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-model-cache\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.761475 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.761457 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-tmp-dir\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.763187 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.763166 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-dshm\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.763425 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.763410 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7102928-5338-4f63-950b-0900b80b07d3-tls-certs\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.818811 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.818770 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qztnq\" (UniqueName: \"kubernetes.io/projected/c7102928-5338-4f63-950b-0900b80b07d3-kube-api-access-qztnq\") pod \"premium-w5-e210ef-kserve-66cccd884b-9m9ck\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:55.951750 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:55.951656 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:56.322978 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:56.322891 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck"] Nov 04 01:13:56.328308 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:13:56.328280 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7102928_5338_4f63_950b_0900b80b07d3.slice/crio-be725acd558a8456901783734c04d80b045c34e2c79b2d99414c4567b5af05bd WatchSource:0}: Error finding container be725acd558a8456901783734c04d80b045c34e2c79b2d99414c4567b5af05bd: Status 404 returned error can't find the container with id be725acd558a8456901783734c04d80b045c34e2c79b2d99414c4567b5af05bd Nov 04 01:13:56.863428 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:56.863390 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" event={"ID":"c7102928-5338-4f63-950b-0900b80b07d3","Type":"ContainerStarted","Data":"c22cb6aa9cd2a7cbb063d334727812a0f14dac19c7fc5f8969582abbfabc60b2"} Nov 04 01:13:56.863428 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:56.863436 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" event={"ID":"c7102928-5338-4f63-950b-0900b80b07d3","Type":"ContainerStarted","Data":"be725acd558a8456901783734c04d80b045c34e2c79b2d99414c4567b5af05bd"} Nov 04 01:13:56.951950 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:56.951898 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:13:56.974105 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:13:56.974053 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podStartSLOduration=1.9740398909999999 podStartE2EDuration="1.974039891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:56.972509578 +0000 UTC m=+982.588855365" watchObservedRunningTime="2026-11-04 01:13:56.974039891 +0000 UTC m=+982.590385654" Nov 04 01:14:05.952225 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:05.952175 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:14:05.965282 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:05.965257 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:14:06.927714 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:06.927688 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:14:15.087650 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:14:15.087565 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:14:21.069080 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.069045 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h"] Nov 04 01:14:21.072791 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.072770 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.172427 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.172397 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c563ee\"/\"premium-w1-c563ee-kserve-self-signed-certs\"" Nov 04 01:14:21.175639 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.175613 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h"] Nov 04 01:14:21.236006 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.235962 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-dshm\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.236216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.236052 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-tmp-dir\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.236216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.236110 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-model-cache\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.236216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.236191 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-home\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.236360 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.236256 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b15931-2a05-4727-9d4a-1c589e9be703-tls-certs\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.236360 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.236291 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gjmd\" (UniqueName: \"kubernetes.io/projected/25b15931-2a05-4727-9d4a-1c589e9be703-kube-api-access-6gjmd\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.337507 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.337418 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-model-cache\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.337507 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.337467 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-home\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.337716 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.337518 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b15931-2a05-4727-9d4a-1c589e9be703-tls-certs\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.337716 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.337550 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gjmd\" (UniqueName: \"kubernetes.io/projected/25b15931-2a05-4727-9d4a-1c589e9be703-kube-api-access-6gjmd\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.337716 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.337660 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-dshm\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.337873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.337722 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-tmp-dir\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.337942 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.337881 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-model-cache\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.338009 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.337954 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-home\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.338082 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.338064 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-tmp-dir\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.339754 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.339731 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-dshm\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.340218 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.340197 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b15931-2a05-4727-9d4a-1c589e9be703-tls-certs\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.394167 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.394110 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gjmd\" (UniqueName: \"kubernetes.io/projected/25b15931-2a05-4727-9d4a-1c589e9be703-kube-api-access-6gjmd\") pod \"premium-w1-c563ee-kserve-77dd759b58-zfs5h\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:21.685326 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:21.685286 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:22.051745 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:22.051656 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h"] Nov 04 01:14:22.057220 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:14:22.057194 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25b15931_2a05_4727_9d4a_1c589e9be703.slice/crio-6e56fbb1c80a309363e0cfa3ca47972af8a1eabd0dbd6df791a07aabcc682892 WatchSource:0}: Error finding container 6e56fbb1c80a309363e0cfa3ca47972af8a1eabd0dbd6df791a07aabcc682892: Status 404 returned error can't find the container with id 6e56fbb1c80a309363e0cfa3ca47972af8a1eabd0dbd6df791a07aabcc682892 Nov 04 01:14:23.030532 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:23.030495 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" event={"ID":"25b15931-2a05-4727-9d4a-1c589e9be703","Type":"ContainerStarted","Data":"adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68"} Nov 04 01:14:23.030996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:23.030541 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" event={"ID":"25b15931-2a05-4727-9d4a-1c589e9be703","Type":"ContainerStarted","Data":"6e56fbb1c80a309363e0cfa3ca47972af8a1eabd0dbd6df791a07aabcc682892"} Nov 04 01:14:23.140805 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:23.140749 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podStartSLOduration=2.140735865 podStartE2EDuration="2.140735865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:23.138494712 +0000 UTC m=+1008.754840476" watchObservedRunningTime="2026-11-04 01:14:23.140735865 +0000 UTC m=+1008.757081655" Nov 04 01:14:23.685725 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:23.685690 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:24.644653 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:24.644617 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:14:31.686062 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:31.686023 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:31.700224 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:31.700197 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:32.089132 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:32.089029 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:14:45.087045 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:14:45.087003 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:14:46.025835 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.025801 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p"] Nov 04 01:14:46.030960 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.030938 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.081865 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.081828 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2fd27d-openshift-default-dockercfg-bkzjb\"" Nov 04 01:14:46.114181 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.114147 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.114181 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.114182 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.114593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.114273 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.114593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.114306 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.114593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.114399 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.114593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.114431 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.114593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.114450 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhqb9\" (UniqueName: \"kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-kube-api-access-qhqb9\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.114593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.114538 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.114784 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.114630 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/705339b0-c4d3-46ad-b55e-919f4113cecc-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.131968 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.131937 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p"] Nov 04 01:14:46.216107 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216072 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhqb9\" (UniqueName: \"kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-kube-api-access-qhqb9\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216107 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216133 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216222 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/705339b0-c4d3-46ad-b55e-919f4113cecc-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216259 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216278 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216320 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216571 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216362 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216571 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216407 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216571 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216443 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216599 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216678 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216740 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216906 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216806 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.216985 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.216965 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/705339b0-c4d3-46ad-b55e-919f4113cecc-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.218573 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.218545 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.218806 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.218788 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.270806 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.270768 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.270952 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.270896 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhqb9\" (UniqueName: \"kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-kube-api-access-qhqb9\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.343807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.343723 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:46.769583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.769544 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p"] Nov 04 01:14:46.775219 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:14:46.775186 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705339b0_c4d3_46ad_b55e_919f4113cecc.slice/crio-b1dee0fc99f8a7836b68c6da52927a6078ad607c97c70d67d00425d1195b66fc WatchSource:0}: Error finding container b1dee0fc99f8a7836b68c6da52927a6078ad607c97c70d67d00425d1195b66fc: Status 404 returned error can't find the container with id b1dee0fc99f8a7836b68c6da52927a6078ad607c97c70d67d00425d1195b66fc Nov 04 01:14:46.777449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.777414 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:14:46.777542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.777497 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:14:46.777598 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:46.777540 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:14:47.169690 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:47.169640 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" event={"ID":"705339b0-c4d3-46ad-b55e-919f4113cecc","Type":"ContainerStarted","Data":"1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72"} Nov 04 01:14:47.170309 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:47.169729 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" event={"ID":"705339b0-c4d3-46ad-b55e-919f4113cecc","Type":"ContainerStarted","Data":"b1dee0fc99f8a7836b68c6da52927a6078ad607c97c70d67d00425d1195b66fc"} Nov 04 01:14:47.331408 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:47.331356 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" podStartSLOduration=2.331342205 podStartE2EDuration="2.331342205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:47.330146841 +0000 UTC m=+1032.946492603" watchObservedRunningTime="2026-11-04 01:14:47.331342205 +0000 UTC m=+1032.947687968" Nov 04 01:14:47.344366 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:47.344336 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:47.345941 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:47.345917 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:14:47.346042 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:47.345973 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" podUID="705339b0-c4d3-46ad-b55e-919f4113cecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 01:14:48.344700 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:48.344664 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:14:48.345103 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:48.344723 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" podUID="705339b0-c4d3-46ad-b55e-919f4113cecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 01:14:49.344583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:49.344545 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:14:49.344761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:49.344603 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" podUID="705339b0-c4d3-46ad-b55e-919f4113cecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 01:14:50.345533 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:50.345492 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:14:50.345996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:50.345557 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" podUID="705339b0-c4d3-46ad-b55e-919f4113cecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 01:14:51.344194 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:51.344154 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:14:51.344402 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:51.344213 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" podUID="705339b0-c4d3-46ad-b55e-919f4113cecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 01:14:52.345036 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:52.344994 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 04 01:14:52.345536 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:52.345060 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" podUID="705339b0-c4d3-46ad-b55e-919f4113cecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 04 01:14:53.483054 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:53.483020 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:53.484185 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:53.484156 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:53.485251 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:53.485227 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:14:57.544300 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:14:57.544266 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:15:00.186967 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.186930 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5"] Nov 04 01:15:00.191091 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.191071 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" Nov 04 01:15:00.196253 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.196230 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btqs\" (UniqueName: \"kubernetes.io/projected/65bd96ef-0f56-4e83-a562-52ed513992e7-kube-api-access-5btqs\") pod \"maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5\" (UID: \"65bd96ef-0f56-4e83-a562-52ed513992e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" Nov 04 01:15:00.196329 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.196282 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65bd96ef-0f56-4e83-a562-52ed513992e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5\" (UID: \"65bd96ef-0f56-4e83-a562-52ed513992e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" Nov 04 01:15:00.241114 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.241080 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:15:00.297591 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.297558 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btqs\" (UniqueName: \"kubernetes.io/projected/65bd96ef-0f56-4e83-a562-52ed513992e7-kube-api-access-5btqs\") pod \"maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5\" (UID: \"65bd96ef-0f56-4e83-a562-52ed513992e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" Nov 04 01:15:00.297761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.297613 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65bd96ef-0f56-4e83-a562-52ed513992e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5\" (UID: \"65bd96ef-0f56-4e83-a562-52ed513992e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" Nov 04 01:15:00.298257 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.298234 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65bd96ef-0f56-4e83-a562-52ed513992e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5\" (UID: \"65bd96ef-0f56-4e83-a562-52ed513992e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" Nov 04 01:15:00.352368 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.352334 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btqs\" (UniqueName: \"kubernetes.io/projected/65bd96ef-0f56-4e83-a562-52ed513992e7-kube-api-access-5btqs\") pod \"maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5\" (UID: \"65bd96ef-0f56-4e83-a562-52ed513992e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" Nov 04 01:15:00.380837 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.380796 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5"] Nov 04 01:15:00.505719 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.505632 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" Nov 04 01:15:00.882995 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.882950 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5"] Nov 04 01:15:00.887928 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:15:00.887891 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bd96ef_0f56_4e83_a562_52ed513992e7.slice/crio-884e8f349e01c2b9e35e170bb951244c8131bfa5f782b02ae0ecb88d2705327f WatchSource:0}: Error finding container 884e8f349e01c2b9e35e170bb951244c8131bfa5f782b02ae0ecb88d2705327f: Status 404 returned error can't find the container with id 884e8f349e01c2b9e35e170bb951244c8131bfa5f782b02ae0ecb88d2705327f Nov 04 01:15:00.890022 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:00.890006 5786 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:15:01.265993 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:01.265901 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" event={"ID":"65bd96ef-0f56-4e83-a562-52ed513992e7","Type":"ContainerStarted","Data":"884e8f349e01c2b9e35e170bb951244c8131bfa5f782b02ae0ecb88d2705327f"} Nov 04 01:15:05.290973 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:05.290930 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" event={"ID":"65bd96ef-0f56-4e83-a562-52ed513992e7","Type":"ContainerStarted","Data":"e441cbc4a2f769940cdf1320bfd1bcf3186b53ebcd5a4fc661287afd1e20b692"} Nov 04 01:15:05.402167 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:05.402088 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" podStartSLOduration=1.37949434 podStartE2EDuration="5.402075222s" totalImagesPullingTime="4.022580882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:05.40133314 +0000 UTC m=+1051.017678903" watchObservedRunningTime="2026-11-04 01:15:05.402075222 +0000 UTC m=+1051.018420985" Nov 04 01:15:06.298658 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:06.298621 5786 generic.go:356] "Generic (PLEG): container finished" podID="65bd96ef-0f56-4e83-a562-52ed513992e7" containerID="e441cbc4a2f769940cdf1320bfd1bcf3186b53ebcd5a4fc661287afd1e20b692" exitCode=60 Nov 04 01:15:06.299064 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:06.298707 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" event={"ID":"65bd96ef-0f56-4e83-a562-52ed513992e7","Type":"ContainerDied","Data":"e441cbc4a2f769940cdf1320bfd1bcf3186b53ebcd5a4fc661287afd1e20b692"} Nov 04 01:15:06.299064 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:06.298931 5786 scope.go:122] "RemoveContainer" containerID="e441cbc4a2f769940cdf1320bfd1bcf3186b53ebcd5a4fc661287afd1e20b692" Nov 04 01:15:07.306295 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:07.306265 5786 generic.go:356] "Generic (PLEG): container finished" podID="65bd96ef-0f56-4e83-a562-52ed513992e7" containerID="e9a83f8ec8eead44b148d74b28ebc339960719d8cbf69853c681b639554a42d1" exitCode=60 Nov 04 01:15:07.306780 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:07.306357 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" event={"ID":"65bd96ef-0f56-4e83-a562-52ed513992e7","Type":"ContainerDied","Data":"e9a83f8ec8eead44b148d74b28ebc339960719d8cbf69853c681b639554a42d1"} Nov 04 01:15:07.306780 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:07.306430 5786 scope.go:122] "RemoveContainer" containerID="e441cbc4a2f769940cdf1320bfd1bcf3186b53ebcd5a4fc661287afd1e20b692" Nov 04 01:15:07.306780 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:07.306710 5786 scope.go:122] "RemoveContainer" containerID="e9a83f8ec8eead44b148d74b28ebc339960719d8cbf69853c681b639554a42d1" Nov 04 01:15:07.307013 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:15:07.306989 5786 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5_odh-ai-gateway-infra(65bd96ef-0f56-4e83-a562-52ed513992e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" Nov 04 01:15:08.314069 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:08.314039 5786 scope.go:122] "RemoveContainer" containerID="e9a83f8ec8eead44b148d74b28ebc339960719d8cbf69853c681b639554a42d1" Nov 04 01:15:08.314494 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:15:08.314318 5786 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5_odh-ai-gateway-infra(65bd96ef-0f56-4e83-a562-52ed513992e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" Nov 04 01:15:10.250500 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:10.250466 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx"] Nov 04 01:15:10.250964 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:10.250675 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" podUID="b98e80b6-dee1-4f43-a02f-c9becae937bd" containerName="istio-proxy" containerID="cri-o://ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18" gracePeriod=30 Nov 04 01:15:14.686664 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.686628 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb"] Nov 04 01:15:14.692177 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.692146 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:14.742173 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.742090 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:15:14.763285 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.763243 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8ptz\" (UniqueName: \"kubernetes.io/projected/f9628c54-8dcb-49df-9404-fe37efa651df-kube-api-access-s8ptz\") pod \"maas-api-revoke-sub-b02b78c8-879bb\" (UID: \"f9628c54-8dcb-49df-9404-fe37efa651df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:14.763448 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.763313 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b02b78c8-879bb\" (UID: \"f9628c54-8dcb-49df-9404-fe37efa651df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:14.763448 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.763375 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b02b78c8-879bb\" (UID: \"f9628c54-8dcb-49df-9404-fe37efa651df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:14.792634 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.792598 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb"] Nov 04 01:15:14.864495 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.864463 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b02b78c8-879bb\" (UID: \"f9628c54-8dcb-49df-9404-fe37efa651df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:14.864664 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.864501 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b02b78c8-879bb\" (UID: \"f9628c54-8dcb-49df-9404-fe37efa651df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:14.864664 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.864584 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8ptz\" (UniqueName: \"kubernetes.io/projected/f9628c54-8dcb-49df-9404-fe37efa651df-kube-api-access-s8ptz\") pod \"maas-api-revoke-sub-b02b78c8-879bb\" (UID: \"f9628c54-8dcb-49df-9404-fe37efa651df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:14.865325 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.865294 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b02b78c8-879bb\" (UID: \"f9628c54-8dcb-49df-9404-fe37efa651df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:14.866767 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.866744 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b02b78c8-879bb\" (UID: \"f9628c54-8dcb-49df-9404-fe37efa651df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:14.919697 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:14.919664 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8ptz\" (UniqueName: \"kubernetes.io/projected/f9628c54-8dcb-49df-9404-fe37efa651df-kube-api-access-s8ptz\") pod \"maas-api-revoke-sub-b02b78c8-879bb\" (UID: \"f9628c54-8dcb-49df-9404-fe37efa651df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:15.006669 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.006583 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:15.096860 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:15:15.096818 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:15:15.377917 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.377891 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb"] Nov 04 01:15:15.384034 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:15:15.384005 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9628c54_8dcb_49df_9404_fe37efa651df.slice/crio-410eefc7e025c9b0cbac22f7727f5a0062c1d8b1760608e26d59da309b29b949 WatchSource:0}: Error finding container 410eefc7e025c9b0cbac22f7727f5a0062c1d8b1760608e26d59da309b29b949: Status 404 returned error can't find the container with id 410eefc7e025c9b0cbac22f7727f5a0062c1d8b1760608e26d59da309b29b949 Nov 04 01:15:15.502822 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.502791 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:15:15.571519 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.571450 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-kube-api-access-pmsxn\" (UniqueName: \"kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-kube-api-access-pmsxn\") pod \"b98e80b6-dee1-4f43-a02f-c9becae937bd\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " Nov 04 01:15:15.571519 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.571504 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-podinfo\") pod \"b98e80b6-dee1-4f43-a02f-c9becae937bd\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " Nov 04 01:15:15.571689 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.571544 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-certs\") pod \"b98e80b6-dee1-4f43-a02f-c9becae937bd\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " Nov 04 01:15:15.571689 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.571562 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-envoy\") pod \"b98e80b6-dee1-4f43-a02f-c9becae937bd\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " Nov 04 01:15:15.571689 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.571592 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-data\") pod \"b98e80b6-dee1-4f43-a02f-c9becae937bd\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " Nov 04 01:15:15.571689 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.571673 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-credential-socket\") pod \"b98e80b6-dee1-4f43-a02f-c9becae937bd\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " Nov 04 01:15:15.571880 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.571704 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-socket\") pod \"b98e80b6-dee1-4f43-a02f-c9becae937bd\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " Nov 04 01:15:15.571880 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.571738 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-token\" (UniqueName: \"kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-token\") pod \"b98e80b6-dee1-4f43-a02f-c9becae937bd\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " Nov 04 01:15:15.571880 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.571780 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98e80b6-dee1-4f43-a02f-c9becae937bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98e80b6-dee1-4f43-a02f-c9becae937bd-istiod-ca-cert\") pod \"b98e80b6-dee1-4f43-a02f-c9becae937bd\" (UID: \"b98e80b6-dee1-4f43-a02f-c9becae937bd\") " Nov 04 01:15:15.572034 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.571897 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-certs" pod "b98e80b6-dee1-4f43-a02f-c9becae937bd" (UID: "b98e80b6-dee1-4f43-a02f-c9becae937bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:15:15.572034 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.571975 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-data" pod "b98e80b6-dee1-4f43-a02f-c9becae937bd" (UID: "b98e80b6-dee1-4f43-a02f-c9becae937bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:15:15.572034 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.571991 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-credential-socket" pod "b98e80b6-dee1-4f43-a02f-c9becae937bd" (UID: "b98e80b6-dee1-4f43-a02f-c9becae937bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:15:15.572232 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.572112 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-socket" pod "b98e80b6-dee1-4f43-a02f-c9becae937bd" (UID: "b98e80b6-dee1-4f43-a02f-c9becae937bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:15:15.572325 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.572303 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98e80b6-dee1-4f43-a02f-c9becae937bd-istiod-ca-cert" pod "b98e80b6-dee1-4f43-a02f-c9becae937bd" (UID: "b98e80b6-dee1-4f43-a02f-c9becae937bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:15.572400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.572388 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:15.572449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.572400 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:15.572449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.572411 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:15.572449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.572419 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:15.572449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.572427 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98e80b6-dee1-4f43-a02f-c9becae937bd-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:15.573856 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.573835 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-podinfo" pod "b98e80b6-dee1-4f43-a02f-c9becae937bd" (UID: "b98e80b6-dee1-4f43-a02f-c9becae937bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:15:15.574030 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.574012 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-token" pod "b98e80b6-dee1-4f43-a02f-c9becae937bd" (UID: "b98e80b6-dee1-4f43-a02f-c9becae937bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:15.574192 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.574161 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-envoy" pod "b98e80b6-dee1-4f43-a02f-c9becae937bd" (UID: "b98e80b6-dee1-4f43-a02f-c9becae937bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:15:15.574291 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.574232 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-kube-api-access-pmsxn" pod "b98e80b6-dee1-4f43-a02f-c9becae937bd" (UID: "b98e80b6-dee1-4f43-a02f-c9becae937bd"). InnerVolumeSpecName "kube-api-access-pmsxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:15.673550 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.673517 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmsxn\" (UniqueName: \"kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-kube-api-access-pmsxn\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:15.673672 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.673557 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:15.673672 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.673573 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:15.673672 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:15.673586 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98e80b6-dee1-4f43-a02f-c9becae937bd-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:16.358159 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:16.358109 5786 generic.go:356] "Generic (PLEG): container finished" podID="b98e80b6-dee1-4f43-a02f-c9becae937bd" containerID="ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18" exitCode=0 Nov 04 01:15:16.358657 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:16.358167 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" Nov 04 01:15:16.358657 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:16.358194 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" event={"ID":"b98e80b6-dee1-4f43-a02f-c9becae937bd","Type":"ContainerDied","Data":"ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18"} Nov 04 01:15:16.358657 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:16.358278 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx" event={"ID":"b98e80b6-dee1-4f43-a02f-c9becae937bd","Type":"ContainerDied","Data":"6652e37a14a1097cd0d6f6bbfbb5e6ef660422fc32c2c34ae9df7f0e3c373826"} Nov 04 01:15:16.358657 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:16.358307 5786 scope.go:122] "RemoveContainer" containerID="ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18" Nov 04 01:15:16.359892 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:16.359871 5786 generic.go:356] "Generic (PLEG): container finished" podID="f9628c54-8dcb-49df-9404-fe37efa651df" containerID="89fb6ac7e3a01a0054676bd8800e287fe1245f27ad1679b135863f07f1952d60" exitCode=0 Nov 04 01:15:16.359995 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:16.359929 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" event={"ID":"f9628c54-8dcb-49df-9404-fe37efa651df","Type":"ContainerDied","Data":"89fb6ac7e3a01a0054676bd8800e287fe1245f27ad1679b135863f07f1952d60"} Nov 04 01:15:16.359995 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:16.359961 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" event={"ID":"f9628c54-8dcb-49df-9404-fe37efa651df","Type":"ContainerStarted","Data":"410eefc7e025c9b0cbac22f7727f5a0062c1d8b1760608e26d59da309b29b949"} Nov 04 01:15:16.368942 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:16.368922 5786 scope.go:122] "RemoveContainer" containerID="ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18" Nov 04 01:15:16.369213 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:15:16.369195 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18\": container with ID starting with ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18 not found: ID does not exist" containerID="ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18" Nov 04 01:15:16.369267 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:16.369220 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18"} err="failed to get container status \"ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18\": rpc error: code = NotFound desc = could not find container \"ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18\": container with ID starting with ecb137514e017ebd34831abae3a0f4bcb02adae73ad994f1e92adfafe4190b18 not found: ID does not exist" Nov 04 01:15:16.630264 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:16.630218 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx"] Nov 04 01:15:16.633639 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:16.633608 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dk4wx"] Nov 04 01:15:17.522630 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:17.522605 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:17.595671 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:17.595628 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-xks-ca\") pod \"f9628c54-8dcb-49df-9404-fe37efa651df\" (UID: \"f9628c54-8dcb-49df-9404-fe37efa651df\") " Nov 04 01:15:17.595889 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:17.595708 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-openshift-ca\") pod \"f9628c54-8dcb-49df-9404-fe37efa651df\" (UID: \"f9628c54-8dcb-49df-9404-fe37efa651df\") " Nov 04 01:15:17.595889 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:17.595783 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9628c54-8dcb-49df-9404-fe37efa651df-kube-api-access-s8ptz\" (UniqueName: \"kubernetes.io/projected/f9628c54-8dcb-49df-9404-fe37efa651df-kube-api-access-s8ptz\") pod \"f9628c54-8dcb-49df-9404-fe37efa651df\" (UID: \"f9628c54-8dcb-49df-9404-fe37efa651df\") " Nov 04 01:15:17.596060 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:17.596031 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-openshift-ca" pod "f9628c54-8dcb-49df-9404-fe37efa651df" (UID: "f9628c54-8dcb-49df-9404-fe37efa651df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:17.596321 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:17.596294 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-openshift-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:17.598004 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:17.597975 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-xks-ca" pod "f9628c54-8dcb-49df-9404-fe37efa651df" (UID: "f9628c54-8dcb-49df-9404-fe37efa651df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:17.598175 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:17.598157 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9628c54-8dcb-49df-9404-fe37efa651df-kube-api-access-s8ptz" pod "f9628c54-8dcb-49df-9404-fe37efa651df" (UID: "f9628c54-8dcb-49df-9404-fe37efa651df"). InnerVolumeSpecName "kube-api-access-s8ptz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:17.697667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:17.697583 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8ptz\" (UniqueName: \"kubernetes.io/projected/f9628c54-8dcb-49df-9404-fe37efa651df-kube-api-access-s8ptz\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:17.697667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:17.697615 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9628c54-8dcb-49df-9404-fe37efa651df-maas-api-xks-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:18.372871 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:18.372841 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" Nov 04 01:15:18.372871 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:18.372867 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb" event={"ID":"f9628c54-8dcb-49df-9404-fe37efa651df","Type":"ContainerDied","Data":"410eefc7e025c9b0cbac22f7727f5a0062c1d8b1760608e26d59da309b29b949"} Nov 04 01:15:18.373091 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:18.372909 5786 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="410eefc7e025c9b0cbac22f7727f5a0062c1d8b1760608e26d59da309b29b949" Nov 04 01:15:18.506147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:18.506088 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98e80b6-dee1-4f43-a02f-c9becae937bd" path="/var/lib/kubelet/pods/b98e80b6-dee1-4f43-a02f-c9becae937bd/volumes" Nov 04 01:15:19.498400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:19.498366 5786 scope.go:122] "RemoveContainer" containerID="e9a83f8ec8eead44b148d74b28ebc339960719d8cbf69853c681b639554a42d1" Nov 04 01:15:20.385343 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:20.385309 5786 generic.go:356] "Generic (PLEG): container finished" podID="65bd96ef-0f56-4e83-a562-52ed513992e7" containerID="33dd84bf8ccf27fdace96650f0b1d9c2c859f425315b9df97d362fb2a648b26a" exitCode=60 Nov 04 01:15:20.385508 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:20.385383 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" event={"ID":"65bd96ef-0f56-4e83-a562-52ed513992e7","Type":"ContainerDied","Data":"33dd84bf8ccf27fdace96650f0b1d9c2c859f425315b9df97d362fb2a648b26a"} Nov 04 01:15:20.385508 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:20.385451 5786 scope.go:122] "RemoveContainer" containerID="e9a83f8ec8eead44b148d74b28ebc339960719d8cbf69853c681b639554a42d1" Nov 04 01:15:20.385810 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:20.385792 5786 scope.go:122] "RemoveContainer" containerID="33dd84bf8ccf27fdace96650f0b1d9c2c859f425315b9df97d362fb2a648b26a" Nov 04 01:15:20.386094 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:15:20.386072 5786 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5_odh-ai-gateway-infra(65bd96ef-0f56-4e83-a562-52ed513992e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" Nov 04 01:15:21.505591 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:21.505559 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5"] Nov 04 01:15:21.666046 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:21.666021 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" Nov 04 01:15:21.741059 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:21.741022 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bd96ef-0f56-4e83-a562-52ed513992e7-kube-api-access-5btqs\" (UniqueName: \"kubernetes.io/projected/65bd96ef-0f56-4e83-a562-52ed513992e7-kube-api-access-5btqs\") pod \"65bd96ef-0f56-4e83-a562-52ed513992e7\" (UID: \"65bd96ef-0f56-4e83-a562-52ed513992e7\") " Nov 04 01:15:21.741282 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:21.741201 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bd96ef-0f56-4e83-a562-52ed513992e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65bd96ef-0f56-4e83-a562-52ed513992e7-maas-api-service-ca\") pod \"65bd96ef-0f56-4e83-a562-52ed513992e7\" (UID: \"65bd96ef-0f56-4e83-a562-52ed513992e7\") " Nov 04 01:15:21.741561 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:21.741539 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bd96ef-0f56-4e83-a562-52ed513992e7-maas-api-service-ca" pod "65bd96ef-0f56-4e83-a562-52ed513992e7" (UID: "65bd96ef-0f56-4e83-a562-52ed513992e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:21.743221 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:21.743200 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bd96ef-0f56-4e83-a562-52ed513992e7-kube-api-access-5btqs" pod "65bd96ef-0f56-4e83-a562-52ed513992e7" (UID: "65bd96ef-0f56-4e83-a562-52ed513992e7"). InnerVolumeSpecName "kube-api-access-5btqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:21.842810 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:21.842726 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65bd96ef-0f56-4e83-a562-52ed513992e7-maas-api-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:21.842810 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:21.842760 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5btqs\" (UniqueName: \"kubernetes.io/projected/65bd96ef-0f56-4e83-a562-52ed513992e7-kube-api-access-5btqs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:22.401415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:22.401375 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" event={"ID":"65bd96ef-0f56-4e83-a562-52ed513992e7","Type":"ContainerDied","Data":"884e8f349e01c2b9e35e170bb951244c8131bfa5f782b02ae0ecb88d2705327f"} Nov 04 01:15:22.401590 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:22.401414 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5" Nov 04 01:15:22.401590 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:22.401452 5786 scope.go:122] "RemoveContainer" containerID="33dd84bf8ccf27fdace96650f0b1d9c2c859f425315b9df97d362fb2a648b26a" Nov 04 01:15:22.564472 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:22.564437 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5"] Nov 04 01:15:22.592205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:22.592172 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895915-jb2l5"] Nov 04 01:15:24.508406 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:24.508370 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" path="/var/lib/kubelet/pods/65bd96ef-0f56-4e83-a562-52ed513992e7/volumes" Nov 04 01:15:40.254905 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.254874 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv"] Nov 04 01:15:40.255373 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.255359 5786 cpu_manager.go:414] "containerMap: removing container" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" containerName="cleanup" Nov 04 01:15:40.255418 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.255374 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" containerName="cleanup" Nov 04 01:15:40.255418 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.255384 5786 cpu_manager.go:414] "containerMap: removing container" podUID="b98e80b6-dee1-4f43-a02f-c9becae937bd" containerName="istio-proxy" Nov 04 01:15:40.255418 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.255389 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98e80b6-dee1-4f43-a02f-c9becae937bd" containerName="istio-proxy" Nov 04 01:15:40.255418 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.255397 5786 cpu_manager.go:414] "containerMap: removing container" podUID="f9628c54-8dcb-49df-9404-fe37efa651df" containerName="revoke-keys" Nov 04 01:15:40.255418 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.255402 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9628c54-8dcb-49df-9404-fe37efa651df" containerName="revoke-keys" Nov 04 01:15:40.255568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.255429 5786 cpu_manager.go:414] "containerMap: removing container" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" containerName="cleanup" Nov 04 01:15:40.255568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.255434 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" containerName="cleanup" Nov 04 01:15:40.255568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.255513 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" containerName="cleanup" Nov 04 01:15:40.255568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.255523 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" containerName="cleanup" Nov 04 01:15:40.255568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.255532 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98e80b6-dee1-4f43-a02f-c9becae937bd" containerName="istio-proxy" Nov 04 01:15:40.255568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.255537 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9628c54-8dcb-49df-9404-fe37efa651df" containerName="revoke-keys" Nov 04 01:15:40.259781 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.259758 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.302758 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.302724 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n"] Nov 04 01:15:40.303262 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.303248 5786 cpu_manager.go:414] "containerMap: removing container" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" containerName="cleanup" Nov 04 01:15:40.303313 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.303263 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" containerName="cleanup" Nov 04 01:15:40.303363 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.303353 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bd96ef-0f56-4e83-a562-52ed513992e7" containerName="cleanup" Nov 04 01:15:40.305625 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.305611 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.341361 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341323 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.341560 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341373 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blvqn\" (UniqueName: \"kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-kube-api-access-blvqn\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.341560 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341464 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9lgq\" (UniqueName: \"kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-kube-api-access-m9lgq\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.341560 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341512 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.341686 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341595 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.341686 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341628 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.341686 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341648 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.341846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341752 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.341846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341815 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfc1116-8951-4304-b1d6-a97825e50c19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.341951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341843 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.341951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341922 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.342056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341965 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.342056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.341997 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.342138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.342092 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.342223 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.342145 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.342280 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.342243 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.342330 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.342277 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.342367 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.342355 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.354066 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.354037 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv"] Nov 04 01:15:40.409917 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.409884 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n"] Nov 04 01:15:40.443659 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.443624 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.443842 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.443676 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.443842 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.443719 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.443842 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.443771 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.443842 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.443806 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blvqn\" (UniqueName: \"kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-kube-api-access-blvqn\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.444065 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.443851 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9lgq\" (UniqueName: \"kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-kube-api-access-m9lgq\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.444065 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.443883 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.444065 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.443922 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.444065 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.443951 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.444065 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.443982 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.444065 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444021 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.444411 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444087 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfc1116-8951-4304-b1d6-a97825e50c19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.444411 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444113 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.444411 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444170 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.444411 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444216 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.444411 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444246 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.444411 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444277 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.444411 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444301 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.444795 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444442 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.444795 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444566 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.444795 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444607 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.444795 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444728 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.445011 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.444902 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.445871 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.445305 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.445871 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.445367 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.445871 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.445554 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfc1116-8951-4304-b1d6-a97825e50c19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.445871 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.445800 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.445871 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.445847 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.447768 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.447743 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.447971 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.447955 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.449075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.449053 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.449202 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.449151 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.501424 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.501361 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.501935 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.501899 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9lgq\" (UniqueName: \"kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-kube-api-access-m9lgq\") pod \"maas-default-gateway-openshift-default-694db55c87-qmg2n\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.502821 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.502794 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blvqn\" (UniqueName: \"kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-kube-api-access-blvqn\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.508886 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.508824 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-2dpsv\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.572012 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.571978 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:40.617091 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.617049 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:40.993666 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:40.993635 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv"] Nov 04 01:15:41.037600 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:15:41.037563 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfc1116_8951_4304_b1d6_a97825e50c19.slice/crio-6cd1d5dc0c2c0e60db6eca58db0962ab1116ce2947e6e8e92288e030e12b00fa WatchSource:0}: Error finding container 6cd1d5dc0c2c0e60db6eca58db0962ab1116ce2947e6e8e92288e030e12b00fa: Status 404 returned error can't find the container with id 6cd1d5dc0c2c0e60db6eca58db0962ab1116ce2947e6e8e92288e030e12b00fa Nov 04 01:15:41.040024 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.039987 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:15:41.040134 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.040076 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:15:41.040134 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.040110 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:15:41.045842 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.045816 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n"] Nov 04 01:15:41.056263 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.056229 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:15:41.056356 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.056297 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:15:41.056356 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.056322 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:15:41.519218 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.519092 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" event={"ID":"fdfc1116-8951-4304-b1d6-a97825e50c19","Type":"ContainerStarted","Data":"c27008e99f4673981079a80e7ffb6a9f51fcd84338cc880510697dc4e1bb1f0e"} Nov 04 01:15:41.519218 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.519182 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" event={"ID":"fdfc1116-8951-4304-b1d6-a97825e50c19","Type":"ContainerStarted","Data":"6cd1d5dc0c2c0e60db6eca58db0962ab1116ce2947e6e8e92288e030e12b00fa"} Nov 04 01:15:41.520886 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.520859 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" event={"ID":"cad8d0d6-46c6-4df3-ae85-50ce778aa84c","Type":"ContainerStarted","Data":"ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb"} Nov 04 01:15:41.521014 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.520912 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" event={"ID":"cad8d0d6-46c6-4df3-ae85-50ce778aa84c","Type":"ContainerStarted","Data":"0b9be40cf4ad5d454f9431944d296fabb8cf3e1bfc62f59c0e2a39ceff093678"} Nov 04 01:15:41.572963 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.572932 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:41.618242 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.618207 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:41.633771 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.633702 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" podStartSLOduration=1.633681954 podStartE2EDuration="1.633681954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:41.631074708 +0000 UTC m=+1087.247420472" watchObservedRunningTime="2026-11-04 01:15:41.633681954 +0000 UTC m=+1087.250027715" Nov 04 01:15:41.747798 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:41.747745 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" podStartSLOduration=1.74773085 podStartE2EDuration="1.74773085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:41.744997083 +0000 UTC m=+1087.361342843" watchObservedRunningTime="2026-11-04 01:15:41.74773085 +0000 UTC m=+1087.364076613" Nov 04 01:15:42.590145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:42.589238 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2dpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:15:42.590145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:42.589307 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" podUID="fdfc1116-8951-4304-b1d6-a97825e50c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:15:42.618944 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:42.618908 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:15:42.619111 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:42.618960 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:15:43.577145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:43.574603 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2dpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:15:43.577145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:43.574668 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" podUID="fdfc1116-8951-4304-b1d6-a97825e50c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:15:43.617712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:43.617670 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:15:43.618101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:43.617739 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:15:44.586200 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:44.582569 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2dpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:15:44.586200 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:44.582636 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" podUID="fdfc1116-8951-4304-b1d6-a97825e50c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:15:44.617999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:44.617964 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:15:44.618432 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:44.618014 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:15:45.134705 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:15:45.134674 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:15:45.577203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:45.574610 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2dpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:15:45.577203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:45.574679 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" podUID="fdfc1116-8951-4304-b1d6-a97825e50c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:15:45.617442 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:45.617405 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:15:45.617586 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:45.617458 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:15:46.604142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:46.582548 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2dpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:15:46.605091 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:46.605057 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" podUID="fdfc1116-8951-4304-b1d6-a97825e50c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:15:46.618276 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:46.618227 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:15:46.618615 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:46.618587 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:15:47.572624 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.572584 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2dpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:15:47.572819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.572646 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" podUID="fdfc1116-8951-4304-b1d6-a97825e50c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:15:47.617610 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.617567 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:15:47.618009 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.617662 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:15:47.859519 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.859477 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8"] Nov 04 01:15:47.862647 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.862623 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:47.912652 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.912608 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e65b23-openshift-default-dockercfg-h6b8n\"" Nov 04 01:15:47.964228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.964189 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8"] Nov 04 01:15:47.976966 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.976926 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:47.976966 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.976963 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:47.977218 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.977095 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptvdn\" (UniqueName: \"kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-kube-api-access-ptvdn\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:47.977218 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.977160 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:47.977306 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.977253 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:47.977351 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.977339 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:47.977385 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.977358 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:47.977431 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.977398 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:47.977547 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:47.977526 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.079092 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.079058 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.079092 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.079097 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.079383 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.079171 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptvdn\" (UniqueName: \"kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-kube-api-access-ptvdn\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.079383 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.079195 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.079383 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.079220 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.079383 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.079358 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.079589 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.079388 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.079589 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.079454 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.079589 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.079523 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.079805 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.079660 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.079873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.079829 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.080062 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.079974 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.080062 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.080005 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.080243 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.080157 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.082656 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.082632 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.082810 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.082790 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.133941 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.133867 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.134097 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.134072 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptvdn\" (UniqueName: \"kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-kube-api-access-ptvdn\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.177084 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.177052 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:48.573577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.573475 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2dpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:15:48.573577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.573538 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" podUID="fdfc1116-8951-4304-b1d6-a97825e50c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:15:48.609045 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.609003 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8"] Nov 04 01:15:48.613993 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:15:48.613961 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c26c24_a4ba_4ec2_9b68_e38872fa1726.slice/crio-d64660be0b29e584713463acffbd1437b9e27675cb6ee59ccd38dd491d3a8fab WatchSource:0}: Error finding container d64660be0b29e584713463acffbd1437b9e27675cb6ee59ccd38dd491d3a8fab: Status 404 returned error can't find the container with id d64660be0b29e584713463acffbd1437b9e27675cb6ee59ccd38dd491d3a8fab Nov 04 01:15:48.617103 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.617067 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:15:48.617242 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.617169 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:15:48.617242 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.617200 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:15:48.617771 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.617752 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qmg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:15:48.618083 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:48.617789 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:15:49.582088 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:49.582049 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" event={"ID":"b9c26c24-a4ba-4ec2-9b68-e38872fa1726","Type":"ContainerStarted","Data":"595fdf925fe7979826a2ddeee1f740d49404bbe3d7e688cffb389d64a4cf6903"} Nov 04 01:15:49.589446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:49.589413 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" event={"ID":"b9c26c24-a4ba-4ec2-9b68-e38872fa1726","Type":"ContainerStarted","Data":"d64660be0b29e584713463acffbd1437b9e27675cb6ee59ccd38dd491d3a8fab"} Nov 04 01:15:49.751090 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:49.751019 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" podStartSLOduration=2.750998987 podStartE2EDuration="2.750998987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:49.750568712 +0000 UTC m=+1095.366914480" watchObservedRunningTime="2026-11-04 01:15:49.750998987 +0000 UTC m=+1095.367344750" Nov 04 01:15:49.956972 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:49.956920 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:49.959082 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:49.959048 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:49.965650 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:49.965607 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:15:49.995607 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:49.995563 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:49.996865 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:49.996845 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:50.001557 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:50.000706 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:15:50.182143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:50.177833 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:50.186138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:50.183079 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:15:50.186138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:50.183169 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:15:51.186289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:51.186239 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:15:51.186700 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:51.186334 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:15:52.185140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:52.181551 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:15:52.185140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:52.181627 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:15:53.186544 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:53.186500 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:15:53.186972 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:53.186604 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:15:54.180595 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:54.180562 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:15:54.180874 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:54.180843 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:15:55.178092 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:55.178053 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:15:55.178556 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:55.178109 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:15:55.660370 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:55.660338 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:15:56.182426 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:56.182396 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:56.182915 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:56.182893 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:15:56.183761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:15:56.183739 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:16:15.089719 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:16:15.089678 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:16:16.018176 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.018141 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7"] Nov 04 01:16:16.023497 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.023475 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.117299 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.117266 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.117739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.117324 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.117739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.117388 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.117739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.117521 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.117739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.117567 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjwkl\" (UniqueName: \"kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-kube-api-access-vjwkl\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.117739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.117623 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.117739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.117648 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.117739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.117690 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.117958 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.117747 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.122619 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.122599 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e210ef-openshift-default-dockercfg-qv9pj\"" Nov 04 01:16:16.124909 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.124890 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7"] Nov 04 01:16:16.218660 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.218624 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.218836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.218668 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.218836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.218724 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.218836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.218792 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.218836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.218830 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.219056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.218847 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.219056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.218879 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.219056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.218928 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjwkl\" (UniqueName: \"kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-kube-api-access-vjwkl\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.219056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.218969 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.219290 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.219102 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.219290 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.219172 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.219290 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.219224 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.219410 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.219302 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.219728 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.219701 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.221097 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.221080 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.221519 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.221504 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.273559 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.273495 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjwkl\" (UniqueName: \"kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-kube-api-access-vjwkl\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.273559 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.273501 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.336463 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.336436 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:16.759288 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.759249 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7"] Nov 04 01:16:16.764500 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:16:16.764474 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a38ad2e_6b50_457c_874b_b4b7944c63c2.slice/crio-08ec3b7b41d1806c7c6a95c667ecb033ed91779254e5ca670a306044ede0993d WatchSource:0}: Error finding container 08ec3b7b41d1806c7c6a95c667ecb033ed91779254e5ca670a306044ede0993d: Status 404 returned error can't find the container with id 08ec3b7b41d1806c7c6a95c667ecb033ed91779254e5ca670a306044ede0993d Nov 04 01:16:16.766699 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.766666 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:16.766781 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.766733 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:16.766781 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:16.766762 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:17.789980 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:17.789937 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" event={"ID":"3a38ad2e-6b50-457c-874b-b4b7944c63c2","Type":"ContainerStarted","Data":"62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590"} Nov 04 01:16:17.790427 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:17.790013 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" event={"ID":"3a38ad2e-6b50-457c-874b-b4b7944c63c2","Type":"ContainerStarted","Data":"08ec3b7b41d1806c7c6a95c667ecb033ed91779254e5ca670a306044ede0993d"} Nov 04 01:16:18.000346 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:18.000289 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" podStartSLOduration=3.000273629 podStartE2EDuration="3.000273629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:17.951780057 +0000 UTC m=+1123.568125855" watchObservedRunningTime="2026-11-04 01:16:18.000273629 +0000 UTC m=+1123.616619471" Nov 04 01:16:18.337332 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:18.337296 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:18.338913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:18.338890 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:16:18.338978 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:18.338953 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" podUID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 04 01:16:19.336974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:19.336938 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:16:19.337418 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:19.336988 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" podUID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 04 01:16:20.337243 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:20.337197 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:16:20.337777 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:20.337253 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" podUID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 04 01:16:21.349271 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:21.349229 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:16:21.349872 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:21.349838 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" podUID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 04 01:16:22.337508 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:22.337475 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:16:22.337712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:22.337526 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" podUID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 04 01:16:23.337839 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:23.337803 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:16:23.338229 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:23.337856 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" podUID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 04 01:16:24.760106 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:24.760074 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:24.760685 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:24.760664 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:24.761621 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:24.761590 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:16:39.511441 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:39.511407 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:16:45.087129 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:16:45.087082 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:16:46.939074 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:46.939038 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh"] Nov 04 01:16:46.944243 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:46.944222 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:46.999390 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:46.999358 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh"] Nov 04 01:16:47.022708 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.022675 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.022708 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.022709 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.022913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.022754 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.022963 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.022916 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.023007 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.022980 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.023166 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.023138 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.023248 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.023190 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.023248 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.023216 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gfhq\" (UniqueName: \"kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-kube-api-access-2gfhq\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.023357 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.023337 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.124588 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.124550 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.124764 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.124608 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.124764 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.124688 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.124764 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.124721 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.124764 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.124742 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gfhq\" (UniqueName: \"kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-kube-api-access-2gfhq\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.124963 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.124812 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.124963 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.124867 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.124963 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.124898 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.125115 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.125024 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.125115 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.125065 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.125260 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.125241 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.125351 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.125325 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.125482 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.125459 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.125565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.125549 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.126963 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.126941 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.127329 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.127313 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.179710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.179673 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.179897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.179796 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gfhq\" (UniqueName: \"kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-kube-api-access-2gfhq\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.256615 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.256526 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:47.679141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.679084 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh"] Nov 04 01:16:47.685981 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:16:47.685955 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec1b4ed_2b7c_4c22_931a_d8b509afd5d4.slice/crio-1f617400e88979b7c482fa9fe0c8f672acb1e494746bc64d06b884d249d4b18c WatchSource:0}: Error finding container 1f617400e88979b7c482fa9fe0c8f672acb1e494746bc64d06b884d249d4b18c: Status 404 returned error can't find the container with id 1f617400e88979b7c482fa9fe0c8f672acb1e494746bc64d06b884d249d4b18c Nov 04 01:16:47.688260 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.688223 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:47.688355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.688296 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:47.688355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:47.688327 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:48.000538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:48.000413 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" event={"ID":"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4","Type":"ContainerStarted","Data":"0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951"} Nov 04 01:16:48.000538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:48.000484 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" event={"ID":"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4","Type":"ContainerStarted","Data":"1f617400e88979b7c482fa9fe0c8f672acb1e494746bc64d06b884d249d4b18c"} Nov 04 01:16:48.113593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:48.113529 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" podStartSLOduration=2.113516872 podStartE2EDuration="2.113516872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:48.111448972 +0000 UTC m=+1153.727794737" watchObservedRunningTime="2026-11-04 01:16:48.113516872 +0000 UTC m=+1153.729862634" Nov 04 01:16:48.256824 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:48.256739 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:48.258504 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:48.258476 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:16:48.258642 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:48.258523 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:16:49.256905 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:49.256862 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:16:49.257390 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:49.256943 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:16:50.256836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:50.256803 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:16:50.257020 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:50.256858 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:16:51.256918 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:51.256886 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:16:51.257097 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:51.256945 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:16:52.256934 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:52.256893 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:16:52.257148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:52.256977 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:16:53.257510 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:53.257473 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:16:53.257962 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:53.257562 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:16:54.256977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:54.256935 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:16:54.257184 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:54.257004 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:16:55.261833 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:55.261799 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:55.262333 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:55.262194 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:16:55.262877 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:16:55.262861 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:17:02.682214 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.682176 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc"] Nov 04 01:17:02.685939 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.685922 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.736341 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.736310 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c563ee-openshift-default-dockercfg-5mps8\"" Nov 04 01:17:02.749638 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.749608 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnx9z\" (UniqueName: \"kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-kube-api-access-dnx9z\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.749771 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.749648 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.749771 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.749714 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.749771 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.749741 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.749869 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.749772 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8d2819-e753-4b38-864b-49faf969feac-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.749869 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.749808 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.749869 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.749841 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.749964 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.749920 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.749964 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.749947 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8d2819-e753-4b38-864b-49faf969feac-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.786470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.786433 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc"] Nov 04 01:17:02.851195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851165 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.851392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851204 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.851392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851223 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8d2819-e753-4b38-864b-49faf969feac-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.851392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851246 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.851392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851291 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.851392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851346 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.851392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851372 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8d2819-e753-4b38-864b-49faf969feac-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.851722 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851537 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnx9z\" (UniqueName: \"kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-kube-api-access-dnx9z\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.851722 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851578 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.851722 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851641 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.851924 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851899 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.851989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851951 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.851989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.851949 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.852156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.852138 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8d2819-e753-4b38-864b-49faf969feac-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.853534 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.853512 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.853903 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.853886 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8d2819-e753-4b38-864b-49faf969feac-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.906098 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.906069 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:02.906472 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:02.906453 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnx9z\" (UniqueName: \"kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-kube-api-access-dnx9z\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:03.000684 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:03.000598 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:03.425272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:03.425232 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc"] Nov 04 01:17:03.429986 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:17:03.429957 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8d2819_e753_4b38_864b_49faf969feac.slice/crio-45e77d2c7e2a3c4d1ad9cf7ae1d2b1b1f989e7d50e64f2aa310dea70c919a434 WatchSource:0}: Error finding container 45e77d2c7e2a3c4d1ad9cf7ae1d2b1b1f989e7d50e64f2aa310dea70c919a434: Status 404 returned error can't find the container with id 45e77d2c7e2a3c4d1ad9cf7ae1d2b1b1f989e7d50e64f2aa310dea70c919a434 Nov 04 01:17:03.432269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:03.432236 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:03.432354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:03.432301 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:03.432354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:03.432332 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:04.126062 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:04.126026 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" event={"ID":"bb8d2819-e753-4b38-864b-49faf969feac","Type":"ContainerStarted","Data":"59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726"} Nov 04 01:17:04.126477 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:04.126087 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" event={"ID":"bb8d2819-e753-4b38-864b-49faf969feac","Type":"ContainerStarted","Data":"45e77d2c7e2a3c4d1ad9cf7ae1d2b1b1f989e7d50e64f2aa310dea70c919a434"} Nov 04 01:17:04.341282 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:04.341226 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" podStartSLOduration=2.341210725 podStartE2EDuration="2.341210725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:04.291787166 +0000 UTC m=+1169.908132932" watchObservedRunningTime="2026-11-04 01:17:04.341210725 +0000 UTC m=+1169.957556488" Nov 04 01:17:05.001261 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:05.001228 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:05.002392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:05.002365 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:17:05.002487 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:05.002410 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" podUID="bb8d2819-e753-4b38-864b-49faf969feac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:17:06.001859 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:06.001822 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:17:06.002285 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:06.001877 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" podUID="bb8d2819-e753-4b38-864b-49faf969feac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:17:06.686442 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:06.686403 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:17:07.001505 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:07.001420 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:17:07.001505 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:07.001481 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" podUID="bb8d2819-e753-4b38-864b-49faf969feac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:17:08.001500 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:08.001464 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:17:08.002045 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:08.001530 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" podUID="bb8d2819-e753-4b38-864b-49faf969feac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:17:09.001078 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:09.001029 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:17:09.001270 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:09.001097 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" podUID="bb8d2819-e753-4b38-864b-49faf969feac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:17:10.001494 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:10.001450 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:17:10.001882 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:10.001516 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" podUID="bb8d2819-e753-4b38-864b-49faf969feac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:17:11.005775 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:11.005743 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:11.006383 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:11.006145 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:11.006900 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:11.006884 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:17:15.137331 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:17:15.137249 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:17:34.619221 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:34.619186 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:17:34.621015 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:34.620991 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:17:45.087237 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:17:45.087196 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:17:53.996464 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:17:53.996426 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:18:15.118251 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:18:15.118178 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:18:37.067809 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:37.067773 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:18:45.087860 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:18:45.087776 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:18:45.675334 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.675301 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4"] Nov 04 01:18:45.679247 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.679227 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.731132 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.731089 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e993d7\"/\"openshift-service-ca.crt\"" Nov 04 01:18:45.731291 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.731181 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e993d7\"/\"kube-root-ca.crt\"" Nov 04 01:18:45.779265 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.779234 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e993d7\"/\"distinct-2-w0-e993d7-kserve-self-signed-certs\"" Nov 04 01:18:45.779265 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.779242 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e993d7\"/\"default-dockercfg-cljcx\"" Nov 04 01:18:45.781463 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.781440 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4"] Nov 04 01:18:45.792702 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.792675 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-dshm\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.792794 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.792710 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-home\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.792794 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.792736 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-model-cache\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.792928 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.792905 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tmp-dir\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.793052 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.793037 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tls-certs\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.793165 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.793148 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94kvx\" (UniqueName: \"kubernetes.io/projected/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-kube-api-access-94kvx\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.893973 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.893935 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94kvx\" (UniqueName: \"kubernetes.io/projected/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-kube-api-access-94kvx\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.894289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.894036 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-dshm\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.894289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.894061 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-home\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.894289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.894075 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-model-cache\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.894289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.894103 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tmp-dir\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.894289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.894264 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tls-certs\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.894603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.894578 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-model-cache\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.894603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.894596 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-home\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.894708 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.894631 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tmp-dir\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.896703 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.896681 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-dshm\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.896811 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.896783 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tls-certs\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.949095 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.949012 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94kvx\" (UniqueName: \"kubernetes.io/projected/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-kube-api-access-94kvx\") pod \"distinct-2-w0-e993d7-kserve-85576d645c-p2xd4\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:45.991626 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:45.991588 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:46.369290 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:46.369261 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4"] Nov 04 01:18:46.374678 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:18:46.374652 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd8c651e_858d_4560_97a5_d3f7f4ce7a21.slice/crio-35084b0b0ad8c1adaf24da73e095727687c8215d71d0a5cb9d4157135a95b63c WatchSource:0}: Error finding container 35084b0b0ad8c1adaf24da73e095727687c8215d71d0a5cb9d4157135a95b63c: Status 404 returned error can't find the container with id 35084b0b0ad8c1adaf24da73e095727687c8215d71d0a5cb9d4157135a95b63c Nov 04 01:18:46.762406 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:46.762304 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" event={"ID":"cd8c651e-858d-4560-97a5-d3f7f4ce7a21","Type":"ContainerStarted","Data":"8cdf6ea600735d12704bf95b2ea53517941a7f7866b037140f4a659ed1f3128a"} Nov 04 01:18:46.762406 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:46.762359 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" event={"ID":"cd8c651e-858d-4560-97a5-d3f7f4ce7a21","Type":"ContainerStarted","Data":"35084b0b0ad8c1adaf24da73e095727687c8215d71d0a5cb9d4157135a95b63c"} Nov 04 01:18:46.876749 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:46.876693 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podStartSLOduration=1.876679787 podStartE2EDuration="1.876679787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:46.876094471 +0000 UTC m=+1272.492440233" watchObservedRunningTime="2026-11-04 01:18:46.876679787 +0000 UTC m=+1272.493025604" Nov 04 01:18:46.992672 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:46.992636 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:55.992028 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:55.991974 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:56.005254 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:56.005221 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:18:56.838027 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:18:56.837951 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:19:15.090428 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:19:15.090384 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:19:19.284114 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:19.284071 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:19:28.519365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.519331 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt"] Nov 04 01:19:28.523163 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.523142 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.536843 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.536816 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-tmp-dir\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.536965 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.536855 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-model-cache\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.537081 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.537044 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d846bbb-bee7-4247-ad29-726123c73ed8-tls-certs\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.537163 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.537144 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-home\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.537225 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.537187 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-dshm\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.537263 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.537226 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v7nm\" (UniqueName: \"kubernetes.io/projected/6d846bbb-bee7-4247-ad29-726123c73ed8-kube-api-access-2v7nm\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.623855 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.623814 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-678aaf\"/\"kube-root-ca.crt\"" Nov 04 01:19:28.624022 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.623874 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-678aaf\"/\"openshift-service-ca.crt\"" Nov 04 01:19:28.624022 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.624007 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-678aaf\"/\"default-dockercfg-wmh77\"" Nov 04 01:19:28.624114 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.624102 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-678aaf\"/\"distinct-2-w6-678aaf-kserve-self-signed-certs\"" Nov 04 01:19:28.626301 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.626280 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt"] Nov 04 01:19:28.637846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.637824 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-dshm\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.637919 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.637858 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v7nm\" (UniqueName: \"kubernetes.io/projected/6d846bbb-bee7-4247-ad29-726123c73ed8-kube-api-access-2v7nm\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.637919 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.637901 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-tmp-dir\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.638022 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.637927 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-model-cache\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.638074 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.638024 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d846bbb-bee7-4247-ad29-726123c73ed8-tls-certs\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.638074 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.638067 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-home\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.638406 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.638382 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-tmp-dir\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.638489 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.638392 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-model-cache\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.638489 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.638451 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-home\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.640112 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.640093 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-dshm\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.640539 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.640518 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d846bbb-bee7-4247-ad29-726123c73ed8-tls-certs\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.726225 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.726194 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v7nm\" (UniqueName: \"kubernetes.io/projected/6d846bbb-bee7-4247-ad29-726123c73ed8-kube-api-access-2v7nm\") pod \"distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:28.835271 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:28.835193 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:29.211852 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:29.211812 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt"] Nov 04 01:19:29.216798 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:19:29.216768 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d846bbb_bee7_4247_ad29_726123c73ed8.slice/crio-b596533dfd7d8795a59e444b90988396268c6af228b288f9bc50493bded5740c WatchSource:0}: Error finding container b596533dfd7d8795a59e444b90988396268c6af228b288f9bc50493bded5740c: Status 404 returned error can't find the container with id b596533dfd7d8795a59e444b90988396268c6af228b288f9bc50493bded5740c Nov 04 01:19:30.029986 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:30.029950 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" event={"ID":"6d846bbb-bee7-4247-ad29-726123c73ed8","Type":"ContainerStarted","Data":"50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8"} Nov 04 01:19:30.029986 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:30.029993 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" event={"ID":"6d846bbb-bee7-4247-ad29-726123c73ed8","Type":"ContainerStarted","Data":"b596533dfd7d8795a59e444b90988396268c6af228b288f9bc50493bded5740c"} Nov 04 01:19:30.141896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:30.141844 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podStartSLOduration=2.141830416 podStartE2EDuration="2.141830416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:30.140930971 +0000 UTC m=+1315.757276731" watchObservedRunningTime="2026-11-04 01:19:30.141830416 +0000 UTC m=+1315.758176175" Nov 04 01:19:30.835766 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:30.835730 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:38.836086 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:38.836044 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:38.848763 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:38.848737 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:39.091614 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:39.091528 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:19:45.120162 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:19:45.120098 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:19:51.790847 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.790812 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9"] Nov 04 01:19:51.795294 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.795271 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:51.893654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.893609 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e65b23\"/\"kube-root-ca.crt\"" Nov 04 01:19:51.893886 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.893733 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e65b23\"/\"openshift-service-ca.crt\"" Nov 04 01:19:51.894019 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.894003 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e65b23\"/\"default-dockercfg-fjq9w\"" Nov 04 01:19:51.894147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.894080 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e65b23\"/\"distinct-2-w4-e65b23-kserve-self-signed-certs\"" Nov 04 01:19:51.897129 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.897102 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9"] Nov 04 01:19:51.988989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.988941 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-dshm\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:51.990236 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.989430 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-tmp-dir\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:51.990236 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.989906 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84fxv\" (UniqueName: \"kubernetes.io/projected/388313a1-0083-4499-96eb-d8acd96280ed-kube-api-access-84fxv\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:51.990236 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.990085 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-home\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:51.990236 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.990146 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388313a1-0083-4499-96eb-d8acd96280ed-tls-certs\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:51.990236 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:51.990183 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-model-cache\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.091409 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.091319 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-home\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.091409 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.091356 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388313a1-0083-4499-96eb-d8acd96280ed-tls-certs\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.091409 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.091380 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-model-cache\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.091693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.091449 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-dshm\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.091693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.091519 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-tmp-dir\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.091693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.091547 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84fxv\" (UniqueName: \"kubernetes.io/projected/388313a1-0083-4499-96eb-d8acd96280ed-kube-api-access-84fxv\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.091867 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.091834 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-model-cache\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.091951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.091926 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-tmp-dir\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.092890 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.092868 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-home\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.093780 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.093758 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-dshm\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.094061 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.094044 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388313a1-0083-4499-96eb-d8acd96280ed-tls-certs\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.146644 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.146608 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84fxv\" (UniqueName: \"kubernetes.io/projected/388313a1-0083-4499-96eb-d8acd96280ed-kube-api-access-84fxv\") pod \"distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.407329 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.407289 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:19:52.829246 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:52.829138 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9"] Nov 04 01:19:52.834022 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:19:52.833993 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod388313a1_0083_4499_96eb_d8acd96280ed.slice/crio-fefd30c299d30f049290b0ff3fbcc16b77aa32f33450b5b785321a2bbdae3c88 WatchSource:0}: Error finding container fefd30c299d30f049290b0ff3fbcc16b77aa32f33450b5b785321a2bbdae3c88: Status 404 returned error can't find the container with id fefd30c299d30f049290b0ff3fbcc16b77aa32f33450b5b785321a2bbdae3c88 Nov 04 01:19:53.172857 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:53.172813 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" event={"ID":"388313a1-0083-4499-96eb-d8acd96280ed","Type":"ContainerStarted","Data":"efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331"} Nov 04 01:19:53.173035 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:53.172866 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" event={"ID":"388313a1-0083-4499-96eb-d8acd96280ed","Type":"ContainerStarted","Data":"fefd30c299d30f049290b0ff3fbcc16b77aa32f33450b5b785321a2bbdae3c88"} Nov 04 01:19:53.284272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:53.284219 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podStartSLOduration=2.284204606 podStartE2EDuration="2.284204606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:53.283859021 +0000 UTC m=+1338.900204784" watchObservedRunningTime="2026-11-04 01:19:53.284204606 +0000 UTC m=+1338.900550368" Nov 04 01:19:53.427499 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:19:53.412822 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:20:02.408134 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:02.408082 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:20:02.420643 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:02.420618 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:20:03.084453 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.084415 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr"] Nov 04 01:20:03.089189 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.089165 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.141646 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.141599 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23550ae-bf0a-44d3-960f-1e10c36b518c-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.141832 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.141708 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvdxz\" (UniqueName: \"kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-kube-api-access-xvdxz\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.141832 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.141744 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.141950 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.141919 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.142003 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.141973 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.142131 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.142090 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.142207 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.142147 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.142282 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.142262 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.142347 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.142303 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.191585 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.191543 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr"] Nov 04 01:20:03.243006 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.242972 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.243265 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.243020 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.243265 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.243061 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23550ae-bf0a-44d3-960f-1e10c36b518c-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.243265 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.243083 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvdxz\" (UniqueName: \"kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-kube-api-access-xvdxz\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.243265 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.243106 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.243265 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.243237 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.243538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.243284 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.243538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.243331 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.243538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.243353 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.243691 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.243626 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.243749 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.243720 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.244012 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.243931 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.244155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.244113 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23550ae-bf0a-44d3-960f-1e10c36b518c-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.244271 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.244247 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.247148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.246057 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.247148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.246252 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.257619 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.257591 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:20:03.298537 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.298488 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.298717 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.298677 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvdxz\" (UniqueName: \"kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-kube-api-access-xvdxz\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.403604 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.403552 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:03.503876 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.503845 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm"] Nov 04 01:20:03.519705 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.519672 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.546916 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.546875 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.547154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.546941 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.547154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.547033 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.547154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.547081 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.547301 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.547224 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpbsx\" (UniqueName: \"kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-kube-api-access-hpbsx\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.547301 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.547280 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.547474 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.547457 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.547585 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.547510 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.547734 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.547713 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.564598 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.564560 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm"] Nov 04 01:20:03.648925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.648882 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649164 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.648934 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649164 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.649047 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649164 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.649090 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649164 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.649158 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.649181 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.649211 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.649307 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbsx\" (UniqueName: \"kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-kube-api-access-hpbsx\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.649331 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649645 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.649619 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649714 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.649690 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.649784 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.649797 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.649911 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.649819 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.651435 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.651412 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.651527 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.651511 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.707880 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.707798 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.708048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.707964 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbsx\" (UniqueName: \"kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-kube-api-access-hpbsx\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:03.832774 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.832734 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr"] Nov 04 01:20:03.837997 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:20:03.837967 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23550ae_bf0a_44d3_960f_1e10c36b518c.slice/crio-5c933daea525cff86ccbeff1726ac4bf98dc6e45e389f109cb61c8f9ce0a7321 WatchSource:0}: Error finding container 5c933daea525cff86ccbeff1726ac4bf98dc6e45e389f109cb61c8f9ce0a7321: Status 404 returned error can't find the container with id 5c933daea525cff86ccbeff1726ac4bf98dc6e45e389f109cb61c8f9ce0a7321 Nov 04 01:20:03.840511 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.840472 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:03.840618 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.840545 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:03.840618 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.840574 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:03.842602 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:03.842397 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:04.259889 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:04.259852 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" event={"ID":"e23550ae-bf0a-44d3-960f-1e10c36b518c","Type":"ContainerStarted","Data":"4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0"} Nov 04 01:20:04.260079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:04.259921 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" event={"ID":"e23550ae-bf0a-44d3-960f-1e10c36b518c","Type":"ContainerStarted","Data":"5c933daea525cff86ccbeff1726ac4bf98dc6e45e389f109cb61c8f9ce0a7321"} Nov 04 01:20:04.286261 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:04.286221 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm"] Nov 04 01:20:04.290855 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:20:04.290828 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e0983c_143e_4cc0_a647_8a1ac8d18ff2.slice/crio-67ba53d2404ce5bb08ae4fdcaa4f3295d5b53a58504402288113fe7a8c7dc50c WatchSource:0}: Error finding container 67ba53d2404ce5bb08ae4fdcaa4f3295d5b53a58504402288113fe7a8c7dc50c: Status 404 returned error can't find the container with id 67ba53d2404ce5bb08ae4fdcaa4f3295d5b53a58504402288113fe7a8c7dc50c Nov 04 01:20:04.293374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:04.293335 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:04.293491 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:04.293418 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:04.293491 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:04.293466 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:04.372957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:04.372913 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" podStartSLOduration=1.372900954 podStartE2EDuration="1.372900954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:04.372131284 +0000 UTC m=+1349.988477042" watchObservedRunningTime="2026-11-04 01:20:04.372900954 +0000 UTC m=+1349.989246717" Nov 04 01:20:04.404260 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:04.404207 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:04.405799 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:04.405765 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 04 01:20:04.405930 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:04.405834 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 04 01:20:05.321420 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:05.320483 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" event={"ID":"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2","Type":"ContainerStarted","Data":"257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62"} Nov 04 01:20:05.321420 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:05.320548 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" event={"ID":"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2","Type":"ContainerStarted","Data":"67ba53d2404ce5bb08ae4fdcaa4f3295d5b53a58504402288113fe7a8c7dc50c"} Nov 04 01:20:05.406142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:05.405600 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 04 01:20:05.406142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:05.405666 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 04 01:20:05.439185 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:05.439127 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" podStartSLOduration=2.439099956 podStartE2EDuration="2.439099956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:05.436375683 +0000 UTC m=+1351.052721448" watchObservedRunningTime="2026-11-04 01:20:05.439099956 +0000 UTC m=+1351.055445796" Nov 04 01:20:05.843992 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:05.843954 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:05.845529 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:05.845506 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:20:05.845691 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:05.845669 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 04 01:20:06.410571 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:06.410534 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 04 01:20:06.411887 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:06.411850 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 04 01:20:06.845216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:06.845183 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:20:06.845376 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:06.845242 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 04 01:20:07.411144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:07.410749 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 04 01:20:07.411144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:07.410826 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 04 01:20:07.845108 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:07.845079 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:20:07.845294 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:07.845192 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 04 01:20:08.405783 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:08.405747 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 04 01:20:08.406132 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:08.406082 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 04 01:20:08.844644 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:08.844614 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:20:08.845011 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:08.844673 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 04 01:20:09.404826 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:09.404786 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 04 01:20:09.405183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:09.405105 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 04 01:20:09.844126 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:09.844080 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:20:09.844285 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:09.844158 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 04 01:20:09.952399 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:09.952368 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:20:10.405134 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:10.405085 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 04 01:20:10.405310 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:10.405191 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 04 01:20:10.851195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:10.847895 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:20:10.851195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:10.847956 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 04 01:20:11.405056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:11.404570 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 04 01:20:11.405056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:11.404651 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 04 01:20:11.853151 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:11.852594 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 04 01:20:11.853151 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:11.852661 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 04 01:20:12.552283 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:12.551214 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 04 01:20:12.690268 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:12.690213 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:12.833334 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:12.833221 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:12.930278 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:12.929221 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:13.112201 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:13.089074 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:13.232175 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:13.232147 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:13.234550 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:13.234526 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:13.243420 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:13.243390 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:20:13.248238 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:13.248196 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:13.479714 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:13.479596 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:13.849562 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:13.849523 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:20:13.849981 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:13.849591 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:14.850478 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:14.850437 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:20:14.850928 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:14.850505 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:14.905675 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:14.905614 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:14.906105 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:14.906079 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:14.906942 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:14.906923 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:20:15.091085 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:20:15.091044 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:20:18.050014 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:18.049977 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb"] Nov 04 01:20:18.052831 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:18.052804 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b02b78c8-879bb"] Nov 04 01:20:18.506056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:18.506019 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9628c54-8dcb-49df-9404-fe37efa651df" path="/var/lib/kubelet/pods/f9628c54-8dcb-49df-9404-fe37efa651df/volumes" Nov 04 01:20:34.317195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.317156 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf"] Nov 04 01:20:34.321899 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.321870 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.423481 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.423443 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf"] Nov 04 01:20:34.478018 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.477983 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.478018 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.478023 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.478262 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.478180 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889517a5-9d9a-4643-9665-dd6b46496379-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.478262 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.478223 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.478401 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.478377 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.478460 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.478422 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdq4n\" (UniqueName: \"kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-kube-api-access-pdq4n\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.478545 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.478527 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.478607 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.478590 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889517a5-9d9a-4643-9665-dd6b46496379-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.478643 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.478627 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.579789 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.579699 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.579789 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.579745 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdq4n\" (UniqueName: \"kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-kube-api-access-pdq4n\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.580033 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.579821 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.580033 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.579881 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889517a5-9d9a-4643-9665-dd6b46496379-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.580177 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.580028 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.580177 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.580139 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.580296 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.580199 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.580296 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.580248 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.580404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.580248 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.580404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.580364 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889517a5-9d9a-4643-9665-dd6b46496379-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.580514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.580410 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.581004 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.580974 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.581109 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.581005 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.581109 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.581053 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889517a5-9d9a-4643-9665-dd6b46496379-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.584151 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.583922 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889517a5-9d9a-4643-9665-dd6b46496379-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.591205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.591167 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.636158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.636106 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdq4n\" (UniqueName: \"kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-kube-api-access-pdq4n\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.636346 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.636157 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:34.935219 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:34.935187 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:35.362095 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:35.362045 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf"] Nov 04 01:20:35.366698 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:20:35.366668 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889517a5_9d9a_4643_9665_dd6b46496379.slice/crio-2e5300cefed94070172aac28875bfae223ae6ce7233b0126f84822d5736ca709 WatchSource:0}: Error finding container 2e5300cefed94070172aac28875bfae223ae6ce7233b0126f84822d5736ca709: Status 404 returned error can't find the container with id 2e5300cefed94070172aac28875bfae223ae6ce7233b0126f84822d5736ca709 Nov 04 01:20:35.369266 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:35.369230 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:35.369353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:35.369299 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:35.369353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:35.369329 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:35.547542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:35.547508 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" event={"ID":"889517a5-9d9a-4643-9665-dd6b46496379","Type":"ContainerStarted","Data":"33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122"} Nov 04 01:20:35.547719 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:35.547562 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" event={"ID":"889517a5-9d9a-4643-9665-dd6b46496379","Type":"ContainerStarted","Data":"2e5300cefed94070172aac28875bfae223ae6ce7233b0126f84822d5736ca709"} Nov 04 01:20:35.663609 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:35.663508 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" podStartSLOduration=1.66349426 podStartE2EDuration="1.66349426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:35.660956132 +0000 UTC m=+1381.277301896" watchObservedRunningTime="2026-11-04 01:20:35.66349426 +0000 UTC m=+1381.279840088" Nov 04 01:20:35.935392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:35.935311 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:36.937439 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:36.937402 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 04 01:20:36.937912 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:36.937466 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" podUID="889517a5-9d9a-4643-9665-dd6b46496379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 04 01:20:37.936347 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:37.936304 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 04 01:20:37.936550 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:37.936381 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" podUID="889517a5-9d9a-4643-9665-dd6b46496379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 04 01:20:38.935572 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:38.935538 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 04 01:20:38.936007 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:38.935590 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" podUID="889517a5-9d9a-4643-9665-dd6b46496379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 04 01:20:39.936169 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:39.936110 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 04 01:20:39.936588 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:39.936191 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" podUID="889517a5-9d9a-4643-9665-dd6b46496379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 04 01:20:40.935952 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:40.935916 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 04 01:20:40.936166 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:40.935978 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" podUID="889517a5-9d9a-4643-9665-dd6b46496379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 04 01:20:41.935916 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:41.935881 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 04 01:20:41.936345 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:41.935942 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" podUID="889517a5-9d9a-4643-9665-dd6b46496379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 04 01:20:42.940757 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:42.940724 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:42.941268 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:42.941144 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:42.942066 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:42.942047 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:20:45.089472 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:20:45.089435 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:20:50.143529 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.143495 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k"] Nov 04 01:20:50.148083 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.148058 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.185048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.185009 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4z4\" (UniqueName: \"kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-kube-api-access-fq4z4\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.185048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.185053 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.185327 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.185178 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.185327 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.185222 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.185397 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.185349 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.185397 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.185377 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.185498 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.185394 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.185498 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.185484 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.185592 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.185580 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.250004 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.249965 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k"] Nov 04 01:20:50.286933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.286890 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.287325 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.287290 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4z4\" (UniqueName: \"kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-kube-api-access-fq4z4\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.287457 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.287351 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.287457 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.287401 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.287457 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.287433 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.287652 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.287532 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.287652 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.287564 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.287652 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.287587 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.287652 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.287648 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.287863 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.287792 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.288061 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.288029 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.288195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.288176 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.288522 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.288499 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.288659 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.288640 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.290264 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.290237 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.290264 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.290253 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.343214 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.343179 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4z4\" (UniqueName: \"kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-kube-api-access-fq4z4\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.343214 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.343202 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.462609 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.462525 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:50.898993 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.898957 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k"] Nov 04 01:20:50.903321 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:20:50.903294 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05dfa65e_31a1_4b7f_9bdf_eb22512fa3c6.slice/crio-c9560de64c7653d40c4e8d641935238dd1940cd9c1eb9bf2f011e33ceed860e6 WatchSource:0}: Error finding container c9560de64c7653d40c4e8d641935238dd1940cd9c1eb9bf2f011e33ceed860e6: Status 404 returned error can't find the container with id c9560de64c7653d40c4e8d641935238dd1940cd9c1eb9bf2f011e33ceed860e6 Nov 04 01:20:50.905980 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.905917 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:50.906105 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.906033 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:50.906105 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:50.906079 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:51.725146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:51.725092 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" event={"ID":"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6","Type":"ContainerStarted","Data":"a90e461a8ce7893a989c8e45fa7ff49b6f5dd72b6787fb771d6968096bb1abe4"} Nov 04 01:20:51.725650 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:51.725203 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" event={"ID":"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6","Type":"ContainerStarted","Data":"c9560de64c7653d40c4e8d641935238dd1940cd9c1eb9bf2f011e33ceed860e6"} Nov 04 01:20:52.463546 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:52.463509 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:52.481304 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:52.481262 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:20:52.481616 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:52.481579 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:20:53.486284 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:53.486239 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:20:53.486721 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:53.486348 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:20:54.486373 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:54.486328 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:20:54.486889 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:54.486399 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:20:55.464585 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:55.464540 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:20:55.464880 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:55.464850 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:20:56.481544 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:56.481502 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:20:56.482183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:56.482146 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:20:57.462964 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:57.462925 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:20:57.463187 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:57.462984 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:20:58.467705 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:58.467675 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:58.468086 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:58.467982 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:58.468779 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:58.468762 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:20:58.644884 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:20:58.644821 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" podStartSLOduration=8.644808068 podStartE2EDuration="8.644808068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:51.933268511 +0000 UTC m=+1397.549614274" watchObservedRunningTime="2026-11-04 01:20:58.644808068 +0000 UTC m=+1404.261153831" Nov 04 01:21:15.088430 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:21:15.088389 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:21:18.129456 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:18.129421 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:21:19.353093 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.353045 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l"] Nov 04 01:21:19.363304 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.363278 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.448153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.439327 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.448153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.439412 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.448153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.439578 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.448153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.439625 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.448153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.439675 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.448153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.439767 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.448153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.439792 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.448153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.439824 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.448153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.439852 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tkhz\" (UniqueName: \"kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-kube-api-access-8tkhz\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.598060 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.578162 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.608709 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.608633 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.630698 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.630658 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.631060 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.631038 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.631311 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.631249 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.631535 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.631514 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.631701 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.631682 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tkhz\" (UniqueName: \"kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-kube-api-access-8tkhz\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.631908 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.631889 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.632161 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.632139 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.633071 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.582842 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l"] Nov 04 01:21:19.681150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.678458 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.696142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.692959 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.716144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.710261 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.716144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.711410 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.716144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.711493 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.819991 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.819906 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tkhz\" (UniqueName: \"kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-kube-api-access-8tkhz\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.869205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.866067 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.871583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.871556 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:19.872031 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:19.872009 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:20.018698 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:20.018401 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:20.756144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:20.746593 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:20.868448 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:20.867239 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:20.923586 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:20.922218 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:21.355425 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:21.355385 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:21:21.376142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:21.372247 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded" Nov 04 01:21:21.379141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:21.376397 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:21.446144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:21.380655 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 04 01:21:21.555143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:21.554203 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 04 01:21:21.853196 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:21.840217 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l"] Nov 04 01:21:21.873310 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:21.873273 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:21:21.874047 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:21.874024 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:21:21.902521 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:21.902422 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:21.902725 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:21.902576 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:21.902725 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:21.902647 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:21.990008 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:21.989875 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:22.003410 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:22.003360 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:21:22.390142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:22.388329 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:22.390142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:22.389361 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:21:22.413143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:22.412574 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:22.474173 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:22.413958 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:21:22.474173 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:22.469844 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:21:22.474173 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:22.470293 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:21:22.582390 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:22.582341 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 04 01:21:22.707216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:22.685177 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:22.765156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:22.685267 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:21:23.246216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.158996 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:21:23.246216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.170838 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:23.246216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.193189 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:21:23.246216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.193664 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:21:23.246216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.196570 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:21:23.246216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.240171 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:23.246216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.241454 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:21:23.246216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.241496 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" event={"ID":"ef17f99d-84f8-4f1c-afbe-266df72a8a99","Type":"ContainerStarted","Data":"6f2333e3a193cbe2e050b5e9402548c8844d2dc3d69541cad074ccf8ba0bb366"} Nov 04 01:21:23.331299 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.331255 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:21:23.471632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.471586 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:23.499393 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.499295 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Nov 04 01:21:23.518889 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.502545 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:21:23.520087 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.502877 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 04 01:21:23.523850 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.504131 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded" Nov 04 01:21:23.652691 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.652663 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:21:23.660827 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.660800 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:21:23.662664 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:23.662639 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:21:24.249369 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:24.249334 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" event={"ID":"ef17f99d-84f8-4f1c-afbe-266df72a8a99","Type":"ContainerStarted","Data":"5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32"} Nov 04 01:21:24.515236 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:24.515110 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" podStartSLOduration=5.515097065 podStartE2EDuration="5.515097065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:24.514370378 +0000 UTC m=+1430.130716141" watchObservedRunningTime="2026-11-04 01:21:24.515097065 +0000 UTC m=+1430.131442828" Nov 04 01:21:25.019444 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:25.019412 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:25.020660 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:25.020634 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:21:25.020766 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:25.020682 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" podUID="ef17f99d-84f8-4f1c-afbe-266df72a8a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 01:21:26.019872 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:26.019827 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:21:26.020405 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:26.019888 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" podUID="ef17f99d-84f8-4f1c-afbe-266df72a8a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 01:21:26.189931 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:26.189888 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv"] Nov 04 01:21:26.190217 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:26.190191 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" podUID="fdfc1116-8951-4304-b1d6-a97825e50c19" containerName="istio-proxy" containerID="cri-o://c27008e99f4673981079a80e7ffb6a9f51fcd84338cc880510697dc4e1bb1f0e" gracePeriod=30 Nov 04 01:21:27.019693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:27.019658 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:21:27.019881 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:27.019715 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" podUID="ef17f99d-84f8-4f1c-afbe-266df72a8a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 01:21:28.019578 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:28.019544 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:21:28.019993 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:28.019627 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" podUID="ef17f99d-84f8-4f1c-afbe-266df72a8a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 01:21:29.019927 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:29.019891 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:21:29.020395 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:29.019950 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" podUID="ef17f99d-84f8-4f1c-afbe-266df72a8a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 01:21:30.020101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:30.020059 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:21:30.020549 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:30.020160 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" podUID="ef17f99d-84f8-4f1c-afbe-266df72a8a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 01:21:31.023817 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.023787 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:31.024250 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.024066 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:31.024723 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.024704 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:21:31.318152 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.317617 5786 generic.go:356] "Generic (PLEG): container finished" podID="fdfc1116-8951-4304-b1d6-a97825e50c19" containerID="c27008e99f4673981079a80e7ffb6a9f51fcd84338cc880510697dc4e1bb1f0e" exitCode=0 Nov 04 01:21:31.318152 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.318044 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" event={"ID":"fdfc1116-8951-4304-b1d6-a97825e50c19","Type":"ContainerDied","Data":"c27008e99f4673981079a80e7ffb6a9f51fcd84338cc880510697dc4e1bb1f0e"} Nov 04 01:21:31.477420 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.477394 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:21:31.623901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.623867 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-envoy\") pod \"fdfc1116-8951-4304-b1d6-a97825e50c19\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " Nov 04 01:21:31.624101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.623948 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-podinfo\") pod \"fdfc1116-8951-4304-b1d6-a97825e50c19\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " Nov 04 01:21:31.624101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.624003 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-data\") pod \"fdfc1116-8951-4304-b1d6-a97825e50c19\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " Nov 04 01:21:31.624101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.624061 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-kube-api-access-blvqn\" (UniqueName: \"kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-kube-api-access-blvqn\") pod \"fdfc1116-8951-4304-b1d6-a97825e50c19\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " Nov 04 01:21:31.624101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.624079 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-token\" (UniqueName: \"kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-token\") pod \"fdfc1116-8951-4304-b1d6-a97825e50c19\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " Nov 04 01:21:31.624353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.624173 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-certs\") pod \"fdfc1116-8951-4304-b1d6-a97825e50c19\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " Nov 04 01:21:31.624353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.624297 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-socket\") pod \"fdfc1116-8951-4304-b1d6-a97825e50c19\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " Nov 04 01:21:31.624353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.624328 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-credential-socket\") pod \"fdfc1116-8951-4304-b1d6-a97825e50c19\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " Nov 04 01:21:31.624515 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.624372 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdfc1116-8951-4304-b1d6-a97825e50c19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfc1116-8951-4304-b1d6-a97825e50c19-istiod-ca-cert\") pod \"fdfc1116-8951-4304-b1d6-a97825e50c19\" (UID: \"fdfc1116-8951-4304-b1d6-a97825e50c19\") " Nov 04 01:21:31.624515 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.624413 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-data" pod "fdfc1116-8951-4304-b1d6-a97825e50c19" (UID: "fdfc1116-8951-4304-b1d6-a97825e50c19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.624687 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.624639 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-credential-socket" pod "fdfc1116-8951-4304-b1d6-a97825e50c19" (UID: "fdfc1116-8951-4304-b1d6-a97825e50c19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.624806 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.624708 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-certs" pod "fdfc1116-8951-4304-b1d6-a97825e50c19" (UID: "fdfc1116-8951-4304-b1d6-a97825e50c19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.624878 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.624846 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-socket" pod "fdfc1116-8951-4304-b1d6-a97825e50c19" (UID: "fdfc1116-8951-4304-b1d6-a97825e50c19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.624970 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.624951 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfc1116-8951-4304-b1d6-a97825e50c19-istiod-ca-cert" pod "fdfc1116-8951-4304-b1d6-a97825e50c19" (UID: "fdfc1116-8951-4304-b1d6-a97825e50c19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:31.625281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.625258 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.625374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.625286 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.625374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.625301 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfc1116-8951-4304-b1d6-a97825e50c19-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.625374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.625315 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.625374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.625329 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.626366 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.626338 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-podinfo" pod "fdfc1116-8951-4304-b1d6-a97825e50c19" (UID: "fdfc1116-8951-4304-b1d6-a97825e50c19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:21:31.626579 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.626555 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-envoy" pod "fdfc1116-8951-4304-b1d6-a97825e50c19" (UID: "fdfc1116-8951-4304-b1d6-a97825e50c19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.626664 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.626651 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-kube-api-access-blvqn" pod "fdfc1116-8951-4304-b1d6-a97825e50c19" (UID: "fdfc1116-8951-4304-b1d6-a97825e50c19"). InnerVolumeSpecName "kube-api-access-blvqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:31.626704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.626660 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-token" pod "fdfc1116-8951-4304-b1d6-a97825e50c19" (UID: "fdfc1116-8951-4304-b1d6-a97825e50c19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:31.726573 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.726526 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.726573 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.726563 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.726573 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.726578 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blvqn\" (UniqueName: \"kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-kube-api-access-blvqn\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.726826 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:31.726591 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfc1116-8951-4304-b1d6-a97825e50c19-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.325292 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:32.325256 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" Nov 04 01:21:32.325292 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:32.325282 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv" event={"ID":"fdfc1116-8951-4304-b1d6-a97825e50c19","Type":"ContainerDied","Data":"6cd1d5dc0c2c0e60db6eca58db0962ab1116ce2947e6e8e92288e030e12b00fa"} Nov 04 01:21:32.325829 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:32.325354 5786 scope.go:122] "RemoveContainer" containerID="c27008e99f4673981079a80e7ffb6a9f51fcd84338cc880510697dc4e1bb1f0e" Nov 04 01:21:32.534603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:32.534540 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv"] Nov 04 01:21:32.539426 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:32.539390 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2dpsv"] Nov 04 01:21:34.509423 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:34.509390 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdfc1116-8951-4304-b1d6-a97825e50c19" path="/var/lib/kubelet/pods/fdfc1116-8951-4304-b1d6-a97825e50c19/volumes" Nov 04 01:21:34.662526 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:34.662496 5786 scope.go:122] "RemoveContainer" containerID="89fb6ac7e3a01a0054676bd8800e287fe1245f27ad1679b135863f07f1952d60" Nov 04 01:21:45.088671 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:21:45.088586 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:21:48.909392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.909357 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz"] Nov 04 01:21:48.910027 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.909932 5786 cpu_manager.go:414] "containerMap: removing container" podUID="fdfc1116-8951-4304-b1d6-a97825e50c19" containerName="istio-proxy" Nov 04 01:21:48.910027 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.909945 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdfc1116-8951-4304-b1d6-a97825e50c19" containerName="istio-proxy" Nov 04 01:21:48.910185 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.910053 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdfc1116-8951-4304-b1d6-a97825e50c19" containerName="istio-proxy" Nov 04 01:21:48.915291 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.915268 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:48.959953 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.959917 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m6fr\" (UniqueName: \"kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-kube-api-access-4m6fr\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:48.960182 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.959970 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:48.960182 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.960105 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:48.960182 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.960171 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:48.960389 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.960216 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:48.960389 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.960270 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:48.960389 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.960295 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:48.960547 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.960443 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:48.960547 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:48.960482 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.024301 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.024264 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz"] Nov 04 01:21:49.062095 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.062043 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6fr\" (UniqueName: \"kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-kube-api-access-4m6fr\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.062350 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.062114 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.062350 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.062170 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.062350 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.062201 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.062350 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.062270 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.062595 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.062562 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.062595 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.062575 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.062702 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.062668 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.062850 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.062828 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.062933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.062853 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.062933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.062905 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.063038 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.062942 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.063347 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.063326 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.064027 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.063996 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.064796 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.064776 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.065274 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.065252 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.118256 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.118221 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.118424 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.118284 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m6fr\" (UniqueName: \"kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-kube-api-access-4m6fr\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.234491 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.234398 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:49.677094 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.677060 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz"] Nov 04 01:21:49.717017 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:21:49.716987 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc18916f_4fd9_4086_a1d5_e8036b0d3165.slice/crio-f4358f2b76d92cfcdda025c4e0cdebde7c685ac2733fe2d4fd63b3d7e8e6e04c WatchSource:0}: Error finding container f4358f2b76d92cfcdda025c4e0cdebde7c685ac2733fe2d4fd63b3d7e8e6e04c: Status 404 returned error can't find the container with id f4358f2b76d92cfcdda025c4e0cdebde7c685ac2733fe2d4fd63b3d7e8e6e04c Nov 04 01:21:49.719404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.719370 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:49.719473 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.719440 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:49.719473 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.719469 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:49.906377 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.906341 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p"] Nov 04 01:21:49.910730 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.910705 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:49.969081 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.968998 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p"] Nov 04 01:21:49.975913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.975878 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:49.976244 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.976216 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b454222-560a-4d7d-8b07-997a983463c7-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:49.976376 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.976269 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:49.976376 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.976299 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:49.976376 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.976331 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:49.976376 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.976366 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b454222-560a-4d7d-8b07-997a983463c7-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:49.976612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.976473 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:49.976612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.976538 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cktlh\" (UniqueName: \"kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-kube-api-access-cktlh\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:49.976612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:49.976568 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.077810 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.077777 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.077959 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.077851 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b454222-560a-4d7d-8b07-997a983463c7-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.077959 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.077872 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.077959 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.077890 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.077959 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.077908 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.077959 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.077936 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b454222-560a-4d7d-8b07-997a983463c7-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.078253 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.078036 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.078253 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.078096 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktlh\" (UniqueName: \"kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-kube-api-access-cktlh\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.078253 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.078162 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.078421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.078267 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.078693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.078664 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.078793 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.078709 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.078793 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.078736 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.079034 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.079014 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b454222-560a-4d7d-8b07-997a983463c7-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.080602 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.080579 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.080684 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.080599 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b454222-560a-4d7d-8b07-997a983463c7-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.133640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.133608 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.133801 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.133780 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cktlh\" (UniqueName: \"kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-kube-api-access-cktlh\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.225091 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.224996 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:50.279428 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.279393 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp"] Nov 04 01:21:50.285432 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.285265 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.338960 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.338917 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp"] Nov 04 01:21:50.382687 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.382641 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.382897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.382719 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.382897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.382780 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4266x\" (UniqueName: \"kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-kube-api-access-4266x\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.382897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.382868 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.383071 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.382938 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.383071 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.382974 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.383071 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.383020 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899e8d41-45fd-4178-95d8-271b97510499-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.383266 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.383105 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.383266 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.383143 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899e8d41-45fd-4178-95d8-271b97510499-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.448093 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.448056 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" event={"ID":"dc18916f-4fd9-4086-a1d5-e8036b0d3165","Type":"ContainerStarted","Data":"c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098"} Nov 04 01:21:50.448320 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.448143 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" event={"ID":"dc18916f-4fd9-4086-a1d5-e8036b0d3165","Type":"ContainerStarted","Data":"f4358f2b76d92cfcdda025c4e0cdebde7c685ac2733fe2d4fd63b3d7e8e6e04c"} Nov 04 01:21:50.493197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.485991 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.493197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.486076 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.493197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.486151 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899e8d41-45fd-4178-95d8-271b97510499-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.493197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.486235 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.493197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.486277 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899e8d41-45fd-4178-95d8-271b97510499-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.493197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.486355 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.493197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.486444 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.493197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.486541 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4266x\" (UniqueName: \"kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-kube-api-access-4266x\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.493197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.486573 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.493197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.487988 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.495486 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.495458 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.496043 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.496017 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899e8d41-45fd-4178-95d8-271b97510499-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.496341 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.496318 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.496604 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.496573 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.499213 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.499184 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.499613 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.499591 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899e8d41-45fd-4178-95d8-271b97510499-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.541889 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.541855 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.550874 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.550846 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4266x\" (UniqueName: \"kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-kube-api-access-4266x\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.607959 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.607924 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:50.613819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.613214 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podStartSLOduration=2.613195627 podStartE2EDuration="2.613195627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:50.588555849 +0000 UTC m=+1456.204901614" watchObservedRunningTime="2026-11-04 01:21:50.613195627 +0000 UTC m=+1456.229541390" Nov 04 01:21:50.613819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.613450 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p"] Nov 04 01:21:50.620370 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:21:50.620346 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b454222_560a_4d7d_8b07_997a983463c7.slice/crio-101c0b58ac4d664fde915b5dab6c1a097b84d2bbc89938d741d64b83208c568d WatchSource:0}: Error finding container 101c0b58ac4d664fde915b5dab6c1a097b84d2bbc89938d741d64b83208c568d: Status 404 returned error can't find the container with id 101c0b58ac4d664fde915b5dab6c1a097b84d2bbc89938d741d64b83208c568d Nov 04 01:21:50.623111 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.623075 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:50.623246 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.623181 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:50.623246 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.623213 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:50.634588 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.634560 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2"] Nov 04 01:21:50.640640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.640618 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.668153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.665946 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw"] Nov 04 01:21:50.673143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.671276 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.694570 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.694530 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.694754 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.694574 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.694754 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.694688 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.694754 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.694733 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.694907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.694758 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.695662 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.695627 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.695783 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.695676 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hpjc\" (UniqueName: \"kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-kube-api-access-7hpjc\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.695783 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.695713 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.695783 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.695739 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.695783 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.695767 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.695998 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.695791 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.695998 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.695832 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699f83a5-7fa4-42be-b125-c11e085ca71e-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.695998 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.695857 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5wf\" (UniqueName: \"kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-kube-api-access-kd5wf\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.695998 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.695889 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.695998 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.695922 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.695998 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.695957 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.695998 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.695980 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.696344 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.696043 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.698828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.698554 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2"] Nov 04 01:21:50.734191 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.726836 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw"] Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797110 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797190 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hpjc\" (UniqueName: \"kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-kube-api-access-7hpjc\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797231 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797262 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797298 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797326 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797374 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699f83a5-7fa4-42be-b125-c11e085ca71e-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797407 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5wf\" (UniqueName: \"kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-kube-api-access-kd5wf\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797443 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797481 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797520 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797547 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797614 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797688 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797714 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797834 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797884 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.797911 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.798148 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.798381 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.798596 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.798843 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.799087 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.799276 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699f83a5-7fa4-42be-b125-c11e085ca71e-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.800075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.799705 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.801512 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.800223 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.801512 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.801246 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.801512 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.801496 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.804066 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.804038 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.805148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.804287 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.805148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.804931 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.806041 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.805999 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.852627 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.852503 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.854241 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.854211 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.854981 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.854955 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5wf\" (UniqueName: \"kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-kube-api-access-kd5wf\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:50.855919 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.855899 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hpjc\" (UniqueName: \"kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-kube-api-access-7hpjc\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:50.997736 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:50.997699 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:51.028307 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.028269 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp"] Nov 04 01:21:51.033846 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:21:51.033820 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899e8d41_45fd_4178_95d8_271b97510499.slice/crio-9528e48ee137f03e9acb1109cfca2423c9c97ab98c6883ae1166a5066997b964 WatchSource:0}: Error finding container 9528e48ee137f03e9acb1109cfca2423c9c97ab98c6883ae1166a5066997b964: Status 404 returned error can't find the container with id 9528e48ee137f03e9acb1109cfca2423c9c97ab98c6883ae1166a5066997b964 Nov 04 01:21:51.036486 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.036452 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:51.036580 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.036518 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:51.036580 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.036547 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:51.044177 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.044137 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:51.236744 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.236713 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:21:51.236851 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.236807 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:21:51.236900 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.236862 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 01:21:51.456025 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.455985 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" event={"ID":"899e8d41-45fd-4178-95d8-271b97510499","Type":"ContainerStarted","Data":"970d375edc0af449a68cd1bf961f32c398c313fa3f3410aeac3b30385d6bc2b8"} Nov 04 01:21:51.456238 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.456051 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" event={"ID":"899e8d41-45fd-4178-95d8-271b97510499","Type":"ContainerStarted","Data":"9528e48ee137f03e9acb1109cfca2423c9c97ab98c6883ae1166a5066997b964"} Nov 04 01:21:51.457639 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.457611 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" event={"ID":"6b454222-560a-4d7d-8b07-997a983463c7","Type":"ContainerStarted","Data":"72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce"} Nov 04 01:21:51.457756 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.457658 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" event={"ID":"6b454222-560a-4d7d-8b07-997a983463c7","Type":"ContainerStarted","Data":"101c0b58ac4d664fde915b5dab6c1a097b84d2bbc89938d741d64b83208c568d"} Nov 04 01:21:51.494586 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.494549 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2"] Nov 04 01:21:51.506254 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:21:51.501362 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699f83a5_7fa4_42be_b125_c11e085ca71e.slice/crio-0cd1eca66d8c11cf2bdec6a5eb11af0f3ec51448f7492aa3d5c28fe31baf461e WatchSource:0}: Error finding container 0cd1eca66d8c11cf2bdec6a5eb11af0f3ec51448f7492aa3d5c28fe31baf461e: Status 404 returned error can't find the container with id 0cd1eca66d8c11cf2bdec6a5eb11af0f3ec51448f7492aa3d5c28fe31baf461e Nov 04 01:21:51.506254 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.503983 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:51.506254 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.504050 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:51.506254 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.504086 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:51.559869 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.551778 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw"] Nov 04 01:21:51.614299 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.610829 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:21:51.634809 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.634716 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podStartSLOduration=1.6346901740000002 podStartE2EDuration="1.634690174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:51.57320181 +0000 UTC m=+1457.189547573" watchObservedRunningTime="2026-11-04 01:21:51.634690174 +0000 UTC m=+1457.251035938" Nov 04 01:21:51.640831 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.636414 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:51.640831 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.636514 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:51.640831 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:51.636564 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:52.230143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:52.229613 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:21:52.233449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:52.233045 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:21:52.233449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:52.233102 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:21:52.240388 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:52.240319 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:21:52.240388 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:52.240365 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 01:21:52.612141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:52.610913 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:21:52.612141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:52.610973 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:21:52.674875 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:52.674825 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" event={"ID":"699f83a5-7fa4-42be-b125-c11e085ca71e","Type":"ContainerStarted","Data":"fb4d659c33c19af3fdf8cdcc6030405e6700122636e173ee9ab49a72b59e6583"} Nov 04 01:21:52.675185 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:52.675158 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" event={"ID":"699f83a5-7fa4-42be-b125-c11e085ca71e","Type":"ContainerStarted","Data":"0cd1eca66d8c11cf2bdec6a5eb11af0f3ec51448f7492aa3d5c28fe31baf461e"} Nov 04 01:21:52.758142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:52.736093 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" event={"ID":"156e3df3-cacc-4b41-9bf5-b2f2d481f57b","Type":"ContainerStarted","Data":"d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c"} Nov 04 01:21:52.758142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:52.748262 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" event={"ID":"156e3df3-cacc-4b41-9bf5-b2f2d481f57b","Type":"ContainerStarted","Data":"28ed4541ef2881a4e78c23656cbb7812add37b24eb62aa3a0107321a0c0492be"} Nov 04 01:21:52.844191 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:52.827082 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podStartSLOduration=3.827057553 podStartE2EDuration="3.827057553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:51.707982341 +0000 UTC m=+1457.324328106" watchObservedRunningTime="2026-11-04 01:21:52.827057553 +0000 UTC m=+1458.443403316" Nov 04 01:21:53.013212 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.012386 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:21:53.013212 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.012450 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:21:53.013212 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.012730 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:21:53.027145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.025634 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podStartSLOduration=3.025596986 podStartE2EDuration="3.025596986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:53.000929375 +0000 UTC m=+1458.617275161" watchObservedRunningTime="2026-11-04 01:21:53.025596986 +0000 UTC m=+1458.641942749" Nov 04 01:21:53.042144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.036263 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podStartSLOduration=3.036238852 podStartE2EDuration="3.036238852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:52.84745282 +0000 UTC m=+1458.463798578" watchObservedRunningTime="2026-11-04 01:21:53.036238852 +0000 UTC m=+1458.652584615" Nov 04 01:21:53.046156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.045523 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:21:53.070143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.049235 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:21:53.070143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.049294 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:21:53.228674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.228640 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:21:53.228946 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.228915 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:21:53.237362 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.237329 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:21:53.237909 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.237879 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 01:21:53.427143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.412028 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.640145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.623817 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:21:53.640145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:53.623893 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:21:54.010152 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.002131 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:21:54.030515 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.030465 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:21:54.063151 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.032914 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.072149 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.049726 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:21:54.079153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.072430 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:21:54.464143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.419201 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.464143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.419776 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:21:54.464143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.453176 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:21:54.464143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.453233 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:21:54.464143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.454251 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:21:54.464143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.454304 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 01:21:54.506336 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.506256 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.628143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.625871 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.628143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.626842 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:21:54.652143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.629196 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.737396 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.737282 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.741289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.741256 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:21:54.741555 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.741525 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:21:54.778822 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.778776 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 04 01:21:54.974753 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:54.974708 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.307143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.306236 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-85cpd" podUID="3a5a107e-19f3-4611-90ad-f345b54e27f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.355237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.308230 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.355237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.320403 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:21:55.355237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.352597 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" podUID="d0506411-64e5-47a9-b8b2-d03f78db8d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded" Nov 04 01:21:55.432291 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.432233 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.567195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.566374 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:21:55.567195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.566439 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:21:55.569245 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.569201 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.639365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.639330 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:21:55.664328 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.664280 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:21:55.665185 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.665158 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:21:55.687254 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.687198 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:21:55.892199 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.666505 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.895344 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.686944 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.901362 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.665363 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.908152 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.744784 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:21:55.957144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.746839 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.957144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.746568 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:21:55.957144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.919572 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:21:55.988107 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:21:55.948108 5786 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.321s" Nov 04 01:21:55.988107 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:55.957536 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-b78686"] Nov 04 01:21:56.026400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.005701 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:21:56.026400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.005761 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:21:56.027222 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.027176 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 01:21:56.028253 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.028232 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:21:56.028483 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.028468 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-b78686"] Nov 04 01:21:56.028622 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.028610 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:21:56.028756 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.028745 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:21:56.028881 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.028870 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:21:56.028994 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.028984 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:21:56.029851 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.029826 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-b78686" Nov 04 01:21:56.047761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.047726 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:21:56.048026 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.048001 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:21:56.151498 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.151409 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bqc6h\"" Nov 04 01:21:56.689206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.305630 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:21:56.689206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.335370 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 01:21:56.689206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.313377 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:21:56.689206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.335443 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:21:56.689206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.572703 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.689206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.572803 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 04 01:21:56.689206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.576968 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5cz\" (UniqueName: \"kubernetes.io/projected/02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4-kube-api-access-px5cz\") pod \"test-curl-7240-b78686\" (UID: \"02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4\") " pod="opendatahub/test-curl-7240-b78686" Nov 04 01:21:56.689206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.578553 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.728188 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.728142 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.736179 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.730190 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": context deadline exceeded" Nov 04 01:21:56.790931 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.764478 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.805692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.772218 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.806515 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.788352 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5cz\" (UniqueName: \"kubernetes.io/projected/02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4-kube-api-access-px5cz\") pod \"test-curl-7240-b78686\" (UID: \"02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4\") " pod="opendatahub/test-curl-7240-b78686" Nov 04 01:21:56.854154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.824225 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:21:56.854154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.851973 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:21:56.895190 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.895144 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.901750 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.901712 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.941017 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.939255 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:21:57.035354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:56.992222 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.035354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.009031 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.035354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.009462 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.035354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.017938 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:21:57.035354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.018018 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:21:57.076143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.046721 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:21:57.076143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.046795 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:21:57.176147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.171705 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:21:57.212971 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.212895 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:21:57.213220 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.213202 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:21:57.214181 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.176955 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5cz\" (UniqueName: \"kubernetes.io/projected/02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4-kube-api-access-px5cz\") pod \"test-curl-7240-b78686\" (UID: \"02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4\") " pod="opendatahub/test-curl-7240-b78686" Nov 04 01:21:57.272147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.249101 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:21:57.272444 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.272404 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 01:21:57.275706 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.275675 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:21:57.275895 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.275874 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:21:57.428469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.428430 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-b78686" Nov 04 01:21:57.591241 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.590347 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.715272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.715224 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:21:57.715879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.715833 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:21:57.716214 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.716189 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.734291 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.734253 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:21:57.832684 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.813835 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.884158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.881472 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.943932 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.943893 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.982178 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.960784 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.982178 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:57.981890 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded" Nov 04 01:21:58.015159 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.010418 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:58.089424 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.078778 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:21:58.089424 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.088289 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:21:58.204636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.204591 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:21:58.204809 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.204655 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:21:58.268143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.239158 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 01:21:58.268143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.239220 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 01:21:58.328006 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.317257 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:21:58.328006 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.317315 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:21:58.595199 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.594042 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:58.623141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.611301 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:21:58.623141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.611362 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:21:58.729269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.729221 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:58.742288 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.734965 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:58.868206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.855530 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:21:58.887552 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:58.882457 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:59.016147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.016085 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:21:59.040626 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.040585 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:21:59.089144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.085988 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 04 01:21:59.120712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.093204 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded" Nov 04 01:21:59.259241 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.257764 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:21:59.259241 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.257821 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:21:59.428430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.269225 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:21:59.428430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.269283 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:21:59.428430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.270491 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:21:59.428430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.270533 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:21:59.615140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.612652 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:59.675355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.656722 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:21:59.675355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.656784 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:21:59.743194 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.739365 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 04 01:21:59.760270 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.748151 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:59.803802 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.790025 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-b78686"] Nov 04 01:21:59.936164 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.924207 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.016438 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:21:59.996000 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02c0f2fb_b9f9_4fa4_8c44_baa11f3a53d4.slice/crio-8226fa49a7fc5077fdeb318f7a30f638121430013e85bf765f7bd35d3353d5c5 WatchSource:0}: Error finding container 8226fa49a7fc5077fdeb318f7a30f638121430013e85bf765f7bd35d3353d5c5: Status 404 returned error can't find the container with id 8226fa49a7fc5077fdeb318f7a30f638121430013e85bf765f7bd35d3353d5c5 Nov 04 01:22:00.016438 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.998982 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.016438 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.999032 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:22:00.016438 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.000479 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:00.032220 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.019944 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.032220 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:21:59.999106 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 04 01:22:00.032636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.032375 5786 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:22:00.117335 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.097754 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.117335 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.110819 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.117335 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.112280 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:22:00.117335 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.114987 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.117335 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.115031 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:22:00.243575 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.243541 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.243857 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.243822 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:22:00.329203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.323401 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:00.329203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.323497 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.443377 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.437692 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-b78686" event={"ID":"02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4","Type":"ContainerStarted","Data":"8226fa49a7fc5077fdeb318f7a30f638121430013e85bf765f7bd35d3353d5c5"} Nov 04 01:22:00.651145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.628305 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.679158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.666220 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.679158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.666281 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:22:00.743270 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.743219 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.748365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.748327 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.939247 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:00.938291 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 04 01:22:01.017154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.003260 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.026809 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.026772 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.034792 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.034756 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:22:01.040208 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.027050 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.056282 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.055523 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.056559 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.056510 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:22:01.103430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.103393 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.115159 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.112234 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.115159 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.112632 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.228731 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.228543 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.228731 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.228606 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:22:01.348667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.348224 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:01.348667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.348289 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.633147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.630857 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.633147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.630926 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:22:01.675549 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.675499 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.746274 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.746224 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.796485 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.752253 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:22:01.957667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:01.957226 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.001151 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.000496 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.001151 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.000981 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:22:02.012348 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.011017 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.046769 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.046727 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.047035 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.047003 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:22:02.062302 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.062252 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.156687 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.129247 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.157257 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.129297 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 04 01:22:02.160580 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.133774 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.254651 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.254614 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.274258 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.274213 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:22:02.361309 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.361265 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:02.361552 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.361529 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.611881 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.611799 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.612410 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.612326 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:22:02.689510 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.689323 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.709145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.702236 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.750339 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.750266 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 04 01:22:02.765235 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.764276 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:22:02.959016 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:02.958969 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 04 01:22:03.010612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.010577 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.010888 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.010846 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:22:03.019198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.015203 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:03.045708 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.045576 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.045708 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.045643 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:22:03.165283 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.165234 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:03.165938 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.165904 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:03.166463 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.166438 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:22:03.221149 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.218463 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:03.374755 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.374255 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:03.374755 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.374331 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:03.461150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.460830 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:03.628206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.611596 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.628206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.611663 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:22:03.704140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.703212 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 04 01:22:03.856325 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.814712 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 04 01:22:03.856325 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.827370 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:03.963304 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:03.963203 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.050158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.026529 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded" Nov 04 01:22:04.050158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.028465 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:04.075179 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.050483 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.075179 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.050542 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:22:04.075179 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.050897 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.075179 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.050936 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:22:04.168299 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.168249 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:22:04.174445 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.174389 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 04 01:22:04.220205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.175195 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:04.256656 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.256618 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:22:04.298556 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.297457 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:04.298556 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.297525 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.298977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.298941 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.432154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.430200 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:04.432154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.430263 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.589400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.589210 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:04.589400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.589271 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.636147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.635135 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.636147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.635196 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:22:04.768318 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.768262 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.847910 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.847869 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 04 01:22:04.848815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.848401 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:22:04.871332 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.871290 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.964077 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:04.964033 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.005659 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.005623 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:22:05.005922 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.005894 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:22:05.029450 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.029402 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.049700 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.049655 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:22:05.050073 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.050031 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:22:05.212206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.184268 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.212206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.184710 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:22:05.222287 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.222239 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.243143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.228243 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:22:05.335146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.310280 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.361150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.342717 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:05.361150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.342782 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.484905 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.484780 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:05.485230 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.485177 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.624385 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.624349 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:22:05.676276 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.676243 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:22:05.676562 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.676531 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 04 01:22:05.787202 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.768821 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.854144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.853473 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.855802 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.855632 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.927657 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.926208 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.927657 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.926338 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.989604 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:05.986204 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.006815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.003773 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:22:06.007669 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.007640 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:22:06.007900 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.007864 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:22:06.053205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.030249 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.054146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.053499 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:22:06.055875 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.055846 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:22:06.056077 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.056050 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:22:06.185362 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.185312 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:22:06.186435 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.185585 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 04 01:22:06.231322 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.231248 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.334230 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.317294 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.430278 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.430234 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:06.433078 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.430552 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.544225 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.544188 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:06.544478 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.544449 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.778344 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.778276 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.870368 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.870313 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.872259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.871739 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 04 01:22:06.981146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.978212 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.999538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.999496 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:22:06.999843 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:06.999808 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:22:07.000047 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.000023 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:07.012914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.012813 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 04 01:22:07.030866 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.030681 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:07.031149 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.031095 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:22:07.049009 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.047540 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:22:07.049009 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.048886 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:22:07.049009 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.048936 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:22:07.185898 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.185854 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:22:07.194250 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.194208 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:07.260780 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.260733 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:07.349007 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.325971 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded" Nov 04 01:22:07.648198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.647552 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:07.648198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.647643 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:07.693144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.691371 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:07.693144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.691483 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:07.782143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.781597 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:07.872393 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.872332 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:22:07.872776 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:07.872717 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.004478 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.004379 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.054160 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.029040 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:22:08.054160 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.029102 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:22:08.054160 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.046763 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:22:08.054160 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.046819 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:22:08.054160 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.047523 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.193259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.193212 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.224138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.208799 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 04 01:22:08.331001 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.330911 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.356138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.356079 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.704413 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.704350 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:08.861144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.782251 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.861144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.828851 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.913149 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.883329 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.913149 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.904567 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:08.913149 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.904662 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.949168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:08.945384 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.023198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.016256 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 01:22:09.023198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.016324 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 01:22:09.054178 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.047387 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:22:09.054178 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.047444 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:22:09.073150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.069246 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.073150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.069665 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.228139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.216197 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:22:09.228139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.220326 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:22:09.228139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.222028 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:22:09.256144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.254095 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:22:09.273830 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.272810 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:22:09.380160 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.355400 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:22:09.380160 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.356377 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.380160 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.373860 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.491155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.489698 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.902139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.876762 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 04 01:22:09.902452 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.892306 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.943545 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.942278 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:09.943545 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:09.942357 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:10.098189 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:10.096082 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:22:10.115135 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:10.113376 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:22:10.190990 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:10.138072 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:22:10.314426 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:10.314394 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:22:10.314765 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:10.314746 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:22:10.349817 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:10.349776 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:22:10.357765 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:10.357725 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:10.460391 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:10.460285 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:22:10.641284 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:10.641246 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-b78686" event={"ID":"02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4","Type":"ContainerStarted","Data":"8621f6d40f93d77489209e1007517bce67c422af08279ddc2394e0c15b7d4935"} Nov 04 01:22:10.648136 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:10.648093 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:22:11.006650 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:11.006466 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:11.006650 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:11.006525 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:11.007484 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:11.007455 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:11.007659 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:11.007633 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:11.151141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:11.148137 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-b78686" podStartSLOduration=14.858861416 podStartE2EDuration="17.148098053s" totalImagesPullingTime="2.289236637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:11.14474556 +0000 UTC m=+1476.761091325" watchObservedRunningTime="2026-11-04 01:22:11.148098053 +0000 UTC m=+1476.764443817" Nov 04 01:22:11.151141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:11.150379 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:22:11.159351 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:11.158258 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:22:11.291201 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:11.290594 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:22:11.295981 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:11.295951 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:22:11.420992 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:11.420959 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:22:11.422904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:11.422877 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:22:12.818503 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:12.818464 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-b78686"] Nov 04 01:22:12.819036 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:12.818672 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-b78686" podUID="02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4" containerName="test-curl-7240-b78686" containerID="cri-o://8621f6d40f93d77489209e1007517bce67c422af08279ddc2394e0c15b7d4935" gracePeriod=2 Nov 04 01:22:12.822271 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:12.822244 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-b78686"] Nov 04 01:22:12.868095 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:22:12.868058 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-b78686\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4" pod="opendatahub/test-curl-7240-b78686" Nov 04 01:22:14.342482 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.335494 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz"] Nov 04 01:22:14.342482 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.336559 5786 cpu_manager.go:414] "containerMap: removing container" podUID="02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4" containerName="test-curl-7240-b78686" Nov 04 01:22:14.342482 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.336592 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4" containerName="test-curl-7240-b78686" Nov 04 01:22:14.342482 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.336861 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4" containerName="test-curl-7240-b78686" Nov 04 01:22:14.350692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.350661 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.418690 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.416599 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-60398d45-openshift-default-dockercfg-jszfr\"" Nov 04 01:22:14.444148 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:22:14.426946 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-b78686\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4" pod="opendatahub/test-curl-7240-b78686" Nov 04 01:22:14.444573 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.432107 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmcnf\" (UniqueName: \"kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-kube-api-access-fmcnf\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.444855 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.444833 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-data\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.444982 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.444966 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-envoy\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.445092 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.445079 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-podinfo\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.445229 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.445213 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-token\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.445328 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.445315 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istiod-ca-cert\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.446259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.445522 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-credential-socket\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.446516 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.446496 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-socket\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.446741 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.446719 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-certs\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.450272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.450251 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz"] Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.585816 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istiod-ca-cert\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.585965 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-credential-socket\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.586083 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-socket\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.586255 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-certs\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.586316 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmcnf\" (UniqueName: \"kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-kube-api-access-fmcnf\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.586440 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-data\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.586476 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-envoy\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.586505 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-podinfo\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.586553 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-token\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.605105 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-credential-socket\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.605461 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-socket\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.605530 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istiod-ca-cert\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.605707 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-certs\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.632914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.605810 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-data\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.636394 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.634602 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-envoy\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.642605 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:22:14.642403 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-b78686\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4" pod="opendatahub/test-curl-7240-b78686" Nov 04 01:22:14.690031 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.689995 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmcnf\" (UniqueName: \"kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-kube-api-access-fmcnf\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.691747 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.691387 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-podinfo\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.708819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.708779 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-token\") pod \"e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:14.758638 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:14.758599 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:15.035214 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.035186 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-b78686" Nov 04 01:22:15.103907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.096690 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4-kube-api-access-px5cz\" (UniqueName: \"kubernetes.io/projected/02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4-kube-api-access-px5cz\") pod \"02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4\" (UID: \"02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4\") " Nov 04 01:22:15.147152 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:22:15.139341 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:22:15.147152 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.140736 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4-kube-api-access-px5cz" pod "02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4" (UID: "02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4"). InnerVolumeSpecName "kube-api-access-px5cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:22:15.202049 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.199237 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px5cz\" (UniqueName: \"kubernetes.io/projected/02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4-kube-api-access-px5cz\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:15.218222 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.218144 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz"] Nov 04 01:22:15.257461 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:22:15.257428 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3031f39_b918_4fa7_bfc4_b23a14bb83d1.slice/crio-29d36faa34712cbe8484c6bfb3376d4e1100453641751b373e08d783a0299f09 WatchSource:0}: Error finding container 29d36faa34712cbe8484c6bfb3376d4e1100453641751b373e08d783a0299f09: Status 404 returned error can't find the container with id 29d36faa34712cbe8484c6bfb3376d4e1100453641751b373e08d783a0299f09 Nov 04 01:22:15.260064 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.260030 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:15.260163 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.260103 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:15.260201 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.260159 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:15.726533 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.726492 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" event={"ID":"f3031f39-b918-4fa7-bfc4-b23a14bb83d1","Type":"ContainerStarted","Data":"a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4"} Nov 04 01:22:15.727020 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.726561 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" event={"ID":"f3031f39-b918-4fa7-bfc4-b23a14bb83d1","Type":"ContainerStarted","Data":"29d36faa34712cbe8484c6bfb3376d4e1100453641751b373e08d783a0299f09"} Nov 04 01:22:15.727865 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.727838 5786 generic.go:356] "Generic (PLEG): container finished" podID="02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4" containerID="8621f6d40f93d77489209e1007517bce67c422af08279ddc2394e0c15b7d4935" exitCode=137 Nov 04 01:22:15.728941 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.728324 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-b78686" Nov 04 01:22:15.728941 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.727933 5786 scope.go:122] "RemoveContainer" containerID="8621f6d40f93d77489209e1007517bce67c422af08279ddc2394e0c15b7d4935" Nov 04 01:22:15.750891 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.750869 5786 scope.go:122] "RemoveContainer" containerID="8621f6d40f93d77489209e1007517bce67c422af08279ddc2394e0c15b7d4935" Nov 04 01:22:15.751248 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:22:15.751227 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8621f6d40f93d77489209e1007517bce67c422af08279ddc2394e0c15b7d4935\": container with ID starting with 8621f6d40f93d77489209e1007517bce67c422af08279ddc2394e0c15b7d4935 not found: ID does not exist" containerID="8621f6d40f93d77489209e1007517bce67c422af08279ddc2394e0c15b7d4935" Nov 04 01:22:15.751317 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.751256 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8621f6d40f93d77489209e1007517bce67c422af08279ddc2394e0c15b7d4935"} err="failed to get container status \"8621f6d40f93d77489209e1007517bce67c422af08279ddc2394e0c15b7d4935\": rpc error: code = NotFound desc = could not find container \"8621f6d40f93d77489209e1007517bce67c422af08279ddc2394e0c15b7d4935\": container with ID starting with 8621f6d40f93d77489209e1007517bce67c422af08279ddc2394e0c15b7d4935 not found: ID does not exist" Nov 04 01:22:15.759922 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.759902 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:15.839386 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:15.839334 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" podStartSLOduration=1.839321084 podStartE2EDuration="1.839321084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:15.838820168 +0000 UTC m=+1481.455165930" watchObservedRunningTime="2026-11-04 01:22:15.839321084 +0000 UTC m=+1481.455666846" Nov 04 01:22:15.887779 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:22:15.887697 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-b78686\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4" pod="opendatahub/test-curl-7240-b78686" Nov 04 01:22:16.506498 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:16.506465 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4" path="/var/lib/kubelet/pods/02c0f2fb-b9f9-4fa4-8c44-baa11f3a53d4/volumes" Nov 04 01:22:16.764886 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:16.764814 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:17.741699 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:17.741668 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:17.742626 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:17.742608 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:22:29.736109 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:29.733367 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg"] Nov 04 01:22:29.747525 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:29.745577 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" Nov 04 01:22:29.772132 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:29.772071 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg"] Nov 04 01:22:29.836141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:29.829404 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-60398d45\"" Nov 04 01:22:29.943153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:29.874598 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-60398d45-dockercfg-bhlm9\"" Nov 04 01:22:29.943153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:29.875760 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4m8w\" (UniqueName: \"kubernetes.io/projected/8c3db980-b5ed-4433-9644-43e066dcc7e7-kube-api-access-p4m8w\") pod \"payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg\" (UID: \"8c3db980-b5ed-4433-9644-43e066dcc7e7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" Nov 04 01:22:29.943153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:29.875817 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3db980-b5ed-4433-9644-43e066dcc7e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg\" (UID: \"8c3db980-b5ed-4433-9644-43e066dcc7e7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" Nov 04 01:22:29.992260 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:29.991180 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4m8w\" (UniqueName: \"kubernetes.io/projected/8c3db980-b5ed-4433-9644-43e066dcc7e7-kube-api-access-p4m8w\") pod \"payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg\" (UID: \"8c3db980-b5ed-4433-9644-43e066dcc7e7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" Nov 04 01:22:29.992260 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:29.991258 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3db980-b5ed-4433-9644-43e066dcc7e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg\" (UID: \"8c3db980-b5ed-4433-9644-43e066dcc7e7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" Nov 04 01:22:29.992526 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:29.992457 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3db980-b5ed-4433-9644-43e066dcc7e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg\" (UID: \"8c3db980-b5ed-4433-9644-43e066dcc7e7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" Nov 04 01:22:30.060259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.033745 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5"] Nov 04 01:22:30.065400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.064334 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5"] Nov 04 01:22:30.065400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.064467 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" Nov 04 01:22:30.086960 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.084726 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4m8w\" (UniqueName: \"kubernetes.io/projected/8c3db980-b5ed-4433-9644-43e066dcc7e7-kube-api-access-p4m8w\") pod \"payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg\" (UID: \"8c3db980-b5ed-4433-9644-43e066dcc7e7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" Nov 04 01:22:30.090890 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.090856 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" Nov 04 01:22:30.096170 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.096111 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8rj\" (UniqueName: \"kubernetes.io/projected/f530be7d-43e4-468d-8078-f208203d0bd0-kube-api-access-sx8rj\") pod \"payload-processing-e2e-derive-60398d45-8df775678-skcn5\" (UID: \"f530be7d-43e4-468d-8078-f208203d0bd0\") " pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" Nov 04 01:22:30.096390 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.096368 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f530be7d-43e4-468d-8078-f208203d0bd0-plugins-config-volume\") pod \"payload-processing-e2e-derive-60398d45-8df775678-skcn5\" (UID: \"f530be7d-43e4-468d-8078-f208203d0bd0\") " pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" Nov 04 01:22:30.197809 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.197765 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f530be7d-43e4-468d-8078-f208203d0bd0-plugins-config-volume\") pod \"payload-processing-e2e-derive-60398d45-8df775678-skcn5\" (UID: \"f530be7d-43e4-468d-8078-f208203d0bd0\") " pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" Nov 04 01:22:30.198012 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.197898 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8rj\" (UniqueName: \"kubernetes.io/projected/f530be7d-43e4-468d-8078-f208203d0bd0-kube-api-access-sx8rj\") pod \"payload-processing-e2e-derive-60398d45-8df775678-skcn5\" (UID: \"f530be7d-43e4-468d-8078-f208203d0bd0\") " pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" Nov 04 01:22:30.198625 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.198600 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f530be7d-43e4-468d-8078-f208203d0bd0-plugins-config-volume\") pod \"payload-processing-e2e-derive-60398d45-8df775678-skcn5\" (UID: \"f530be7d-43e4-468d-8078-f208203d0bd0\") " pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" Nov 04 01:22:30.253400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.253332 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8rj\" (UniqueName: \"kubernetes.io/projected/f530be7d-43e4-468d-8078-f208203d0bd0-kube-api-access-sx8rj\") pod \"payload-processing-e2e-derive-60398d45-8df775678-skcn5\" (UID: \"f530be7d-43e4-468d-8078-f208203d0bd0\") " pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" Nov 04 01:22:30.414048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.414008 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" Nov 04 01:22:30.483498 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.483468 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg"] Nov 04 01:22:30.491748 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:22:30.491714 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3db980_b5ed_4433_9644_43e066dcc7e7.slice/crio-b5e129781913da71cf7bf38305c60ebb8652f15aa4a9c965f13f0a6755bcb2a0 WatchSource:0}: Error finding container b5e129781913da71cf7bf38305c60ebb8652f15aa4a9c965f13f0a6755bcb2a0: Status 404 returned error can't find the container with id b5e129781913da71cf7bf38305c60ebb8652f15aa4a9c965f13f0a6755bcb2a0 Nov 04 01:22:30.791485 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:30.791403 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5"] Nov 04 01:22:30.795921 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:22:30.795892 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf530be7d_43e4_468d_8078_f208203d0bd0.slice/crio-7e081da76f14264bc0cce05d86f41fdfbaa2f76e0f79fae9a25e50a9b3156890 WatchSource:0}: Error finding container 7e081da76f14264bc0cce05d86f41fdfbaa2f76e0f79fae9a25e50a9b3156890: Status 404 returned error can't find the container with id 7e081da76f14264bc0cce05d86f41fdfbaa2f76e0f79fae9a25e50a9b3156890 Nov 04 01:22:31.049130 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:31.049004 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" event={"ID":"f530be7d-43e4-468d-8078-f208203d0bd0","Type":"ContainerStarted","Data":"1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846"} Nov 04 01:22:31.049303 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:31.049112 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" Nov 04 01:22:31.049303 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:31.049173 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" event={"ID":"f530be7d-43e4-468d-8078-f208203d0bd0","Type":"ContainerStarted","Data":"7e081da76f14264bc0cce05d86f41fdfbaa2f76e0f79fae9a25e50a9b3156890"} Nov 04 01:22:31.050801 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:31.050771 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" event={"ID":"8c3db980-b5ed-4433-9644-43e066dcc7e7","Type":"ContainerStarted","Data":"8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930"} Nov 04 01:22:31.050943 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:31.050815 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" event={"ID":"8c3db980-b5ed-4433-9644-43e066dcc7e7","Type":"ContainerStarted","Data":"b5e129781913da71cf7bf38305c60ebb8652f15aa4a9c965f13f0a6755bcb2a0"} Nov 04 01:22:31.050943 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:31.050857 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" Nov 04 01:22:31.165018 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:31.164961 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" podStartSLOduration=2.164946815 podStartE2EDuration="2.164946815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:31.161739931 +0000 UTC m=+1496.778085694" watchObservedRunningTime="2026-11-04 01:22:31.164946815 +0000 UTC m=+1496.781292578" Nov 04 01:22:31.276368 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:31.276306 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" podStartSLOduration=2.276290315 podStartE2EDuration="2.276290315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:31.274900173 +0000 UTC m=+1496.891245937" watchObservedRunningTime="2026-11-04 01:22:31.276290315 +0000 UTC m=+1496.892636107" Nov 04 01:22:34.662250 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:34.662222 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:22:34.664931 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:34.664898 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:22:34.971920 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:34.971821 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp"] Nov 04 01:22:34.979154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:34.979110 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.061028 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.060985 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.061320 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.061295 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.061464 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.061449 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.061685 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.061669 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.061807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.061794 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.061905 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.061891 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.062074 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.062060 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.068142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.062291 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzgj\" (UniqueName: \"kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-kube-api-access-kbzgj\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.068445 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.068413 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.068733 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.063034 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj"] Nov 04 01:22:35.171648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.169394 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.171648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.169439 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.171648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.169459 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.171648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.169513 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.171648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.169595 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzgj\" (UniqueName: \"kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-kube-api-access-kbzgj\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.171648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.169615 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.171648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.169663 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.171648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.169678 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.171648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.169703 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.171648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.170087 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.171648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.170275 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.171648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.170472 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.172449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.172168 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.172449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.172411 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.173032 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.173008 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp"] Nov 04 01:22:35.173172 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.173055 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k"] Nov 04 01:22:35.174352 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.174316 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" Nov 04 01:22:35.197898 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.180302 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj"] Nov 04 01:22:35.197898 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.180427 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" Nov 04 01:22:35.197898 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.180856 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.197898 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.182374 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.253053 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.249862 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e400a3a6\"" Nov 04 01:22:35.281144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.276941 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e400a3a6-dockercfg-d77kv\"" Nov 04 01:22:35.293155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.288871 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f373a64-18f6-4b78-ba0c-dc506a788e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj\" (UID: \"9f373a64-18f6-4b78-ba0c-dc506a788e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" Nov 04 01:22:35.293155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.288942 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kps\" (UniqueName: \"kubernetes.io/projected/9f373a64-18f6-4b78-ba0c-dc506a788e27-kube-api-access-p7kps\") pod \"payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj\" (UID: \"9f373a64-18f6-4b78-ba0c-dc506a788e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" Nov 04 01:22:35.293155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.289269 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc52bbbc-616f-4a2a-be0e-33d9294159bf-plugins-config-volume\") pod \"payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k\" (UID: \"cc52bbbc-616f-4a2a-be0e-33d9294159bf\") " pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" Nov 04 01:22:35.310023 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.289525 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q55tj\" (UniqueName: \"kubernetes.io/projected/cc52bbbc-616f-4a2a-be0e-33d9294159bf-kube-api-access-q55tj\") pod \"payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k\" (UID: \"cc52bbbc-616f-4a2a-be0e-33d9294159bf\") " pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" Nov 04 01:22:35.414189 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.397199 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc52bbbc-616f-4a2a-be0e-33d9294159bf-plugins-config-volume\") pod \"payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k\" (UID: \"cc52bbbc-616f-4a2a-be0e-33d9294159bf\") " pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" Nov 04 01:22:35.430823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.405437 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.430823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.406351 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc52bbbc-616f-4a2a-be0e-33d9294159bf-plugins-config-volume\") pod \"payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k\" (UID: \"cc52bbbc-616f-4a2a-be0e-33d9294159bf\") " pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" Nov 04 01:22:35.431549 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.431517 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q55tj\" (UniqueName: \"kubernetes.io/projected/cc52bbbc-616f-4a2a-be0e-33d9294159bf-kube-api-access-q55tj\") pod \"payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k\" (UID: \"cc52bbbc-616f-4a2a-be0e-33d9294159bf\") " pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" Nov 04 01:22:35.445612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.445573 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f373a64-18f6-4b78-ba0c-dc506a788e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj\" (UID: \"9f373a64-18f6-4b78-ba0c-dc506a788e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" Nov 04 01:22:35.445941 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.445904 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kps\" (UniqueName: \"kubernetes.io/projected/9f373a64-18f6-4b78-ba0c-dc506a788e27-kube-api-access-p7kps\") pod \"payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj\" (UID: \"9f373a64-18f6-4b78-ba0c-dc506a788e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" Nov 04 01:22:35.451495 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.448879 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f373a64-18f6-4b78-ba0c-dc506a788e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj\" (UID: \"9f373a64-18f6-4b78-ba0c-dc506a788e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" Nov 04 01:22:35.456375 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.456339 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k"] Nov 04 01:22:35.499131 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.499084 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q55tj\" (UniqueName: \"kubernetes.io/projected/cc52bbbc-616f-4a2a-be0e-33d9294159bf-kube-api-access-q55tj\") pod \"payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k\" (UID: \"cc52bbbc-616f-4a2a-be0e-33d9294159bf\") " pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" Nov 04 01:22:35.505256 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.505177 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzgj\" (UniqueName: \"kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-kube-api-access-kbzgj\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.508255 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.508087 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kps\" (UniqueName: \"kubernetes.io/projected/9f373a64-18f6-4b78-ba0c-dc506a788e27-kube-api-access-p7kps\") pod \"payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj\" (UID: \"9f373a64-18f6-4b78-ba0c-dc506a788e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" Nov 04 01:22:35.627721 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.627684 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:35.732579 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.732542 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s"] Nov 04 01:22:35.740129 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.740089 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.770548 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.770471 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" Nov 04 01:22:35.778522 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.778492 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" Nov 04 01:22:35.803040 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.803003 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s"] Nov 04 01:22:35.861082 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.861044 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.861251 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.861218 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.861317 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.861299 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.861359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.861351 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzj9d\" (UniqueName: \"kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-kube-api-access-kzj9d\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.861410 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.861399 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.861456 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.861445 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.861497 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.861486 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.861532 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.861520 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.861568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.861545 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.871861 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.871829 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:22:35.962843 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.962805 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.963050 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.962869 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.963050 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.962926 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzj9d\" (UniqueName: \"kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-kube-api-access-kzj9d\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.963050 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.963012 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.963237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.963071 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.963237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.963145 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.963237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.963196 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.963383 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.963233 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.963383 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.963343 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.963709 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.963632 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.963709 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.963635 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.963984 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.963903 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.963984 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.963940 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.964114 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.964048 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.965623 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.965598 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:35.966169 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:35.966147 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:36.019421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.019388 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:36.019576 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.019454 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzj9d\" (UniqueName: \"kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-kube-api-access-kzj9d\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:36.059345 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.059255 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp"] Nov 04 01:22:36.061220 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.061190 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:36.062796 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:22:36.062768 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57ffc7e_b9c6_44e0_a5d3_08df226ee170.slice/crio-3d9494940b6caa42137e84321ac0da9ea2fab631a503973e57326978f63a4149 WatchSource:0}: Error finding container 3d9494940b6caa42137e84321ac0da9ea2fab631a503973e57326978f63a4149: Status 404 returned error can't find the container with id 3d9494940b6caa42137e84321ac0da9ea2fab631a503973e57326978f63a4149 Nov 04 01:22:36.065605 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.065572 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:36.065672 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.065637 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:36.065740 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.065672 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:36.153950 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.144799 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" event={"ID":"f57ffc7e-b9c6-44e0-a5d3-08df226ee170","Type":"ContainerStarted","Data":"3d9494940b6caa42137e84321ac0da9ea2fab631a503973e57326978f63a4149"} Nov 04 01:22:36.165354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.164484 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k"] Nov 04 01:22:36.173375 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.171738 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj"] Nov 04 01:22:36.175868 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:22:36.175839 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f373a64_18f6_4b78_ba0c_dc506a788e27.slice/crio-681492429f2d9b6a0b9652ce3b539dcdb62d6c1fe8fcda3a0529b7674d434b27 WatchSource:0}: Error finding container 681492429f2d9b6a0b9652ce3b539dcdb62d6c1fe8fcda3a0529b7674d434b27: Status 404 returned error can't find the container with id 681492429f2d9b6a0b9652ce3b539dcdb62d6c1fe8fcda3a0529b7674d434b27 Nov 04 01:22:36.508160 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.503147 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s"] Nov 04 01:22:36.535024 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.534970 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:36.535385 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.535354 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:36.535604 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:36.535534 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:37.211433 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.211392 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" event={"ID":"f57ffc7e-b9c6-44e0-a5d3-08df226ee170","Type":"ContainerStarted","Data":"017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e"} Nov 04 01:22:37.217096 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.217045 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" event={"ID":"9f373a64-18f6-4b78-ba0c-dc506a788e27","Type":"ContainerStarted","Data":"01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e"} Nov 04 01:22:37.217336 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.217320 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" event={"ID":"9f373a64-18f6-4b78-ba0c-dc506a788e27","Type":"ContainerStarted","Data":"681492429f2d9b6a0b9652ce3b539dcdb62d6c1fe8fcda3a0529b7674d434b27"} Nov 04 01:22:37.218333 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.218305 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" Nov 04 01:22:37.292873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.287587 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" event={"ID":"cc52bbbc-616f-4a2a-be0e-33d9294159bf","Type":"ContainerStarted","Data":"3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c"} Nov 04 01:22:37.292873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.287664 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" event={"ID":"cc52bbbc-616f-4a2a-be0e-33d9294159bf","Type":"ContainerStarted","Data":"6fcb50a436c5d11452278e2ebb01bef4b22bc2a07e1f87755be1784ee89854b5"} Nov 04 01:22:37.292873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.288759 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" Nov 04 01:22:37.320956 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.320921 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" event={"ID":"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6","Type":"ContainerStarted","Data":"041de15dd17b778b4cd24cfb93805c01067fa510237f85f4fb028017d0adcd69"} Nov 04 01:22:37.321271 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.321234 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" event={"ID":"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6","Type":"ContainerStarted","Data":"8ab4795ea838c74a8d1ed9a12b25c73a9188a745d2c97a5c48e9c9d40c291833"} Nov 04 01:22:37.364148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.362547 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podStartSLOduration=3.362522992 podStartE2EDuration="3.362522992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:37.335014474 +0000 UTC m=+1502.951360240" watchObservedRunningTime="2026-11-04 01:22:37.362522992 +0000 UTC m=+1502.978868756" Nov 04 01:22:37.475075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.470759 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" podStartSLOduration=3.470737855 podStartE2EDuration="3.470737855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:37.453626264 +0000 UTC m=+1503.069972030" watchObservedRunningTime="2026-11-04 01:22:37.470737855 +0000 UTC m=+1503.087083618" Nov 04 01:22:37.618102 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.606316 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" podStartSLOduration=2.606290773 podStartE2EDuration="2.606290773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:37.567738677 +0000 UTC m=+1503.184084445" watchObservedRunningTime="2026-11-04 01:22:37.606290773 +0000 UTC m=+1503.222636537" Nov 04 01:22:37.680429 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.655102 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:37.680429 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.672298 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:37.680429 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.672374 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:37.794586 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:37.772592 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podStartSLOduration=2.772571924 podStartE2EDuration="2.772571924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:37.77050391 +0000 UTC m=+1503.386849675" watchObservedRunningTime="2026-11-04 01:22:37.772571924 +0000 UTC m=+1503.388917690" Nov 04 01:22:38.064000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.062536 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:38.079153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.064403 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:38.079153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.064474 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:38.125144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.116217 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.607175 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.601441 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.656326 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.637562 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:38.656326 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.637643 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:38.831375 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.831331 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.834479 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.834352 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.834727 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.831646 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.896436 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.883316 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.909665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.909286 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:38.909665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:38.909338 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded" Nov 04 01:22:39.065144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.048737 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 04 01:22:39.077144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.071357 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:39.077144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.071419 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:39.126146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.125212 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:22:39.126146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.125661 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:22:39.659412 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.659365 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 04 01:22:39.663139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.661866 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:22:39.910237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.892537 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.962997 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.915566 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.962997 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.915621 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.962997 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.915640 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:39.962997 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.915674 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.962997 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.937217 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 04 01:22:39.962997 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.942213 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.962997 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:39.943309 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:40.100478 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.100440 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:22:40.100854 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.100833 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:22:40.127529 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.108651 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:22:40.127529 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.108713 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:22:40.127529 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.110339 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:22:40.127529 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.110385 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:22:40.127529 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.110931 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:40.142309 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.129196 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:40.172149 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.148438 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:40.172769 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.158932 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:40.172997 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.172961 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:40.173334 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.173317 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:22:40.303305 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.303259 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:40.303652 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.303626 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:40.697392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:40.697316 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:41.403148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.400874 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:41.403148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.400976 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:41.417153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.416501 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 04 01:22:41.429168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.427609 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 04 01:22:41.568143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.567959 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:41.634247 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.634179 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded" Nov 04 01:22:41.736209 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.688961 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:22:41.736209 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.689716 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:41.736209 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.568032 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:41.757404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.757343 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 04 01:22:41.814147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.778335 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:22:41.814693 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:22:41.780161 5786 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.28s" Nov 04 01:22:41.815442 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.781724 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:41.815442 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.815379 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:41.838173 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.638571 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:41.838173 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.828195 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:41.894577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.894531 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" Nov 04 01:22:41.959145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.959036 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:41.959145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:41.959092 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:42.191209 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.112653 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:42.191209 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.112712 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:42.191209 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.136347 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" Nov 04 01:22:42.191209 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.156989 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" Nov 04 01:22:42.444206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.418833 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:42.495146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.444467 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 04 01:22:42.653147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.644770 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:42.653147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.644827 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:42.653147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.644893 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:42.694379 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.694260 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:42.698249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.698214 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:42.698518 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.698482 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" podUID="bb8d2819-e753-4b38-864b-49faf969feac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:42.698687 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.698669 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:42.739982 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.739830 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:42.739982 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.739902 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:42.748144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.746209 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:42.758313 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.758259 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:42.758641 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.758590 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:42.802144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.797216 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 04 01:22:42.864144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.824217 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:42.864144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.825012 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:42.943300 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:42.919345 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:43.126154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.100774 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:43.126709 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.126664 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:43.437205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.436585 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:43.454146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.446572 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 04 01:22:43.454146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.447383 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 04 01:22:43.635068 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.635025 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:43.650931 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.650892 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:43.651294 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.649285 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:43.710494 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.710381 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:43.711526 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.710733 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:43.807517 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.807472 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:43.848020 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.847976 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:43.868145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.849542 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:22:43.928101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.927220 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:43.998506 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.998284 5786 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:43.998506 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:43.998362 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" podUID="889517a5-9d9a-4643-9665-dd6b46496379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.089143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.088799 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:44.089143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.088861 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:44.129263 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.129214 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.302532 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.302438 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:44.302825 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.302797 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.451963 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.451911 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 04 01:22:44.452695 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.452666 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 04 01:22:44.542623 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.542589 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:44.542929 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.542902 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.636674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.636629 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:44.636969 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.636932 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:44.687145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.667067 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.731147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.712102 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.746175 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.732145 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.944206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.812617 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.944206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.860056 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:44.944206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.861028 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.944206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.861141 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:22:44.977145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:44.949086 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.157899 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.097231 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.191326 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.191286 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:45.487326 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.487273 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.514161 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.489815 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 04 01:22:45.749667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.733218 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.775145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.767796 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.775145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.767883 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:45.775145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.768177 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:22:45.775145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.768869 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": context deadline exceeded" Nov 04 01:22:45.822363 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.822290 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.871468 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.871426 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:22:45.872090 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.872060 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:45.928606 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.928555 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Nov 04 01:22:45.942279 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.936445 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.963386 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:45.963324 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.117420 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.117346 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.138167 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.118598 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:46.318017 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.317382 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k" podUID="d0506411-64e5-47a9-b8b2-d03f78db8d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.491285 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.491186 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.567144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.537975 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.633548 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.633514 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.633869 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.633823 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:46.730144 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:22:46.724381 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:22:46.760208 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.757507 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:46.827939 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.817562 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" podUID="0b8dfc4b-7e71-4360-91cd-a22c7685da9e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.827939 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.822872 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 04 01:22:46.849822 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.849778 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.896954 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.879307 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.914796 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.905229 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.971645 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.967676 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.971645 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:46.967785 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.001273 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.001229 5786 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 04 01:22:47.076945 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.076853 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.077216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.077188 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:47.095866 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.095834 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:22:47.154000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.153968 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:22:47.154577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.154556 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:22:47.190203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.190168 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:22:47.240162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.239808 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:22:47.337480 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.337395 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:22:47.343422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.343391 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:22:47.346025 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.345998 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2" Nov 04 01:22:47.348309 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.348282 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:22:47.632142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.628499 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.632142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:47.628560 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:48.070203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:48.068578 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.070203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:48.068647 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:48.628788 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:48.628748 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.629010 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:48.628820 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:49.064219 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:49.062674 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.064219 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:49.062736 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:49.628836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:49.628799 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.629254 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:49.628851 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:50.086203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:50.082570 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.086203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:50.082636 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:50.628679 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:50.628635 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:50.628879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:50.628855 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.629324 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:50.628903 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 04 01:22:50.923589 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:50.923505 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" Nov 04 01:22:51.067163 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:51.066362 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:51.068188 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:51.067759 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.068188 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:51.067807 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 04 01:22:52.551051 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:52.551014 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:52.552767 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:52.552741 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:22:52.669329 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:52.669277 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:52.669531 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:52.669368 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:52.905029 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:52.905000 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:22:52.906024 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:22:52.906007 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:23:04.253954 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.253912 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn"] Nov 04 01:23:04.267430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.267393 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.301017 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.300973 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2"] Nov 04 01:23:04.306980 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.306955 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.320330 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.320229 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.320330 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.320287 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.320577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.320450 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061cb628-1307-41db-b238-e432a72f9150-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.320577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.320506 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.320577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.320532 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmb98\" (UniqueName: \"kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-kube-api-access-dmb98\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.320577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.320564 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061cb628-1307-41db-b238-e432a72f9150-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.320799 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.320710 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.320856 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.320802 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.320856 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.320837 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.340754 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.340650 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn"] Nov 04 01:23:04.411477 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.411389 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2"] Nov 04 01:23:04.433852 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.433814 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.437682 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.437302 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.437682 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.437369 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.437682 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.437523 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.437682 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.437577 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df90e93-d322-4d8e-87b3-74c38bd2e428-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.437682 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.437650 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.438042 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.437695 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.438042 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.437756 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.438042 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.437823 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.438042 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.437864 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061cb628-1307-41db-b238-e432a72f9150-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.438042 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.437892 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtrbx\" (UniqueName: \"kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-kube-api-access-gtrbx\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.438042 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.437949 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.438042 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.437978 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmb98\" (UniqueName: \"kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-kube-api-access-dmb98\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.438042 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.438008 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061cb628-1307-41db-b238-e432a72f9150-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.438574 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.438551 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.438798 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.438778 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.439165 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.439146 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.439342 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.439315 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.439829 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.439164 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.439829 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.439538 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.439829 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.439682 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.440080 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.440061 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.440257 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.440234 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061cb628-1307-41db-b238-e432a72f9150-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.443283 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.443256 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.444195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.444149 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061cb628-1307-41db-b238-e432a72f9150-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.509361 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.509212 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmb98\" (UniqueName: \"kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-kube-api-access-dmb98\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.531893 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.531856 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.541396 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.541354 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.542003 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.541974 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.542171 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.542147 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.542263 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.542193 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.542321 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.542264 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.542321 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.542308 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df90e93-d322-4d8e-87b3-74c38bd2e428-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.542509 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.542360 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.542509 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.542451 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.542509 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.542491 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtrbx\" (UniqueName: \"kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-kube-api-access-gtrbx\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.542698 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.542561 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.543550 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.543176 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.543550 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.543444 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df90e93-d322-4d8e-87b3-74c38bd2e428-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.543550 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.543484 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.543550 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.543525 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.546027 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.546002 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.560479 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.560443 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.606305 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.606237 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:04.607496 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.607465 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtrbx\" (UniqueName: \"kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-kube-api-access-gtrbx\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.610068 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.610036 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:04.641803 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:04.641772 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:05.092285 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.091692 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn"] Nov 04 01:23:05.094294 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:23:05.094265 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061cb628_1307_41db_b238_e432a72f9150.slice/crio-533570ec79dac6fadd1379e24fc3d80f7f36adf10dcf23b130f746420137f6f8 WatchSource:0}: Error finding container 533570ec79dac6fadd1379e24fc3d80f7f36adf10dcf23b130f746420137f6f8: Status 404 returned error can't find the container with id 533570ec79dac6fadd1379e24fc3d80f7f36adf10dcf23b130f746420137f6f8 Nov 04 01:23:05.098518 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.098428 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:05.098658 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.098530 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:05.098658 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.098581 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:05.145078 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.145049 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2"] Nov 04 01:23:05.149068 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:23:05.149042 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df90e93_d322_4d8e_87b3_74c38bd2e428.slice/crio-85bbce46381effe59c7ce42990d33f1c1f61689d503528d0afae23c342bbb323 WatchSource:0}: Error finding container 85bbce46381effe59c7ce42990d33f1c1f61689d503528d0afae23c342bbb323: Status 404 returned error can't find the container with id 85bbce46381effe59c7ce42990d33f1c1f61689d503528d0afae23c342bbb323 Nov 04 01:23:05.152150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.151960 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:05.152150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.152057 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:05.152150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.152102 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:05.299162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.299095 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" event={"ID":"061cb628-1307-41db-b238-e432a72f9150","Type":"ContainerStarted","Data":"be91ab787ebf5234c3000461a552b7a5c76a534a09484a82a992036b3bf3b352"} Nov 04 01:23:05.299662 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.299190 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" event={"ID":"061cb628-1307-41db-b238-e432a72f9150","Type":"ContainerStarted","Data":"533570ec79dac6fadd1379e24fc3d80f7f36adf10dcf23b130f746420137f6f8"} Nov 04 01:23:05.303240 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.303205 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" event={"ID":"4df90e93-d322-4d8e-87b3-74c38bd2e428","Type":"ContainerStarted","Data":"715fde94ffffffb5dc0fa8788e6031402aee074b997a09b04243b76aeb0ddf40"} Nov 04 01:23:05.303453 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.303432 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" event={"ID":"4df90e93-d322-4d8e-87b3-74c38bd2e428","Type":"ContainerStarted","Data":"85bbce46381effe59c7ce42990d33f1c1f61689d503528d0afae23c342bbb323"} Nov 04 01:23:05.443607 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.442555 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" podStartSLOduration=1.442530917 podStartE2EDuration="1.442530917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:05.427418977 +0000 UTC m=+1531.043764743" watchObservedRunningTime="2026-11-04 01:23:05.442530917 +0000 UTC m=+1531.058876682" Nov 04 01:23:05.573142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.571762 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" podStartSLOduration=1.571742025 podStartE2EDuration="1.571742025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:05.560485375 +0000 UTC m=+1531.176831140" watchObservedRunningTime="2026-11-04 01:23:05.571742025 +0000 UTC m=+1531.188087791" Nov 04 01:23:05.616306 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.615256 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:05.643035 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:05.642953 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:06.000796 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.000760 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g"] Nov 04 01:23:06.009111 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.008563 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.087144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.067780 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.087144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.067844 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwcxg\" (UniqueName: \"kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-kube-api-access-dwcxg\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.087144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.068032 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.087144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.068109 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.087144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.068187 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.087144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.068216 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.087144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.068264 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.087144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.068351 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.087144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.068378 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.125183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.115532 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g"] Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.200634 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.200696 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.200781 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.200829 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwcxg\" (UniqueName: \"kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-kube-api-access-dwcxg\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.206643 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.215385 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.215535 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.215631 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.215681 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.215766 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.216427 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.216792 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.217172 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.221156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.218053 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.223001 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.222972 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.224039 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.224019 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.316200 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.310992 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwcxg\" (UniqueName: \"kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-kube-api-access-dwcxg\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.317974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.317937 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.330925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.330893 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:06.613709 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.613665 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:23:06.613896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.613742 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" podUID="061cb628-1307-41db-b238-e432a72f9150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 04 01:23:06.644165 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.644130 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:23:06.644323 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.644184 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 04 01:23:06.819980 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.819868 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g"] Nov 04 01:23:06.823835 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:23:06.823808 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ca1b1e_0ca9_485f_8f2b_1d013b618b66.slice/crio-bdfbd0e79eccea9c940aeec3278eac4e2be7ebbab2e91528583ccfb95a89aacc WatchSource:0}: Error finding container bdfbd0e79eccea9c940aeec3278eac4e2be7ebbab2e91528583ccfb95a89aacc: Status 404 returned error can't find the container with id bdfbd0e79eccea9c940aeec3278eac4e2be7ebbab2e91528583ccfb95a89aacc Nov 04 01:23:06.827496 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.827455 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:06.827568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.827541 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:06.827620 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:06.827580 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:07.333560 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:07.333468 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" event={"ID":"95ca1b1e-0ca9-485f-8f2b-1d013b618b66","Type":"ContainerStarted","Data":"bfc49586d872509d524d1a106fb72b90a9dcb19c6469dcea49937c43d2a13e19"} Nov 04 01:23:07.334028 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:07.333570 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" event={"ID":"95ca1b1e-0ca9-485f-8f2b-1d013b618b66","Type":"ContainerStarted","Data":"bdfbd0e79eccea9c940aeec3278eac4e2be7ebbab2e91528583ccfb95a89aacc"} Nov 04 01:23:07.501143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:07.498310 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" podStartSLOduration=2.498286578 podStartE2EDuration="2.498286578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:07.458820251 +0000 UTC m=+1533.075166013" watchObservedRunningTime="2026-11-04 01:23:07.498286578 +0000 UTC m=+1533.114632334" Nov 04 01:23:07.612145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:07.611662 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:23:07.612145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:07.611727 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" podUID="061cb628-1307-41db-b238-e432a72f9150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 04 01:23:07.643853 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:07.642906 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:23:07.643853 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:07.642960 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 04 01:23:08.339356 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.339259 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:08.340023 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.340001 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:23:08.340240 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.340205 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" podUID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 04 01:23:08.411258 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.411218 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv"] Nov 04 01:23:08.445213 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.444976 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:08.460191 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.460151 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv"] Nov 04 01:23:08.516148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.515408 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-60398d45-serving-cert\"" Nov 04 01:23:08.710141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.709839 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:23:08.710141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.709899 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" podUID="061cb628-1307-41db-b238-e432a72f9150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 04 01:23:08.763147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.711553 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:23:08.763147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.711599 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 04 01:23:08.900238 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.830406 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-maas-api-tls\") pod \"maas-api-e2e-derive-60398d45-64776c497c-sqfvv\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:08.900238 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.830462 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-openshift-service-ca\") pod \"maas-api-e2e-derive-60398d45-64776c497c-sqfvv\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:08.900238 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.830548 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-metrics-certs\") pod \"maas-api-e2e-derive-60398d45-64776c497c-sqfvv\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:08.900238 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.830626 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4phsw\" (UniqueName: \"kubernetes.io/projected/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-kube-api-access-4phsw\") pod \"maas-api-e2e-derive-60398d45-64776c497c-sqfvv\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:08.977272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.961208 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4phsw\" (UniqueName: \"kubernetes.io/projected/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-kube-api-access-4phsw\") pod \"maas-api-e2e-derive-60398d45-64776c497c-sqfvv\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:08.977272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.974134 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-maas-api-tls\") pod \"maas-api-e2e-derive-60398d45-64776c497c-sqfvv\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:08.977272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.974222 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-openshift-service-ca\") pod \"maas-api-e2e-derive-60398d45-64776c497c-sqfvv\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:08.977272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:08.974384 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-metrics-certs\") pod \"maas-api-e2e-derive-60398d45-64776c497c-sqfvv\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:09.013687 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:09.007150 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-metrics-certs\") pod \"maas-api-e2e-derive-60398d45-64776c497c-sqfvv\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:09.023466 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:09.012207 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-openshift-service-ca\") pod \"maas-api-e2e-derive-60398d45-64776c497c-sqfvv\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:09.123146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:09.120973 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4phsw\" (UniqueName: \"kubernetes.io/projected/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-kube-api-access-4phsw\") pod \"maas-api-e2e-derive-60398d45-64776c497c-sqfvv\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:09.138141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:09.134806 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-maas-api-tls\") pod \"maas-api-e2e-derive-60398d45-64776c497c-sqfvv\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:09.157813 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:09.157408 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:09.332145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:09.331629 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:23:09.332145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:09.331690 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" podUID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 04 01:23:09.612142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:09.611301 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:23:09.612142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:09.611361 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" podUID="061cb628-1307-41db-b238-e432a72f9150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 04 01:23:09.652707 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:09.652651 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:23:09.652914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:09.652730 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 04 01:23:10.039168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:10.036624 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv"] Nov 04 01:23:10.042870 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:23:10.042747 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6e5ba9_6335_4740_b5d8_1c371fca6d97.slice/crio-f9b0e125dd557a8af827c2b8fc7bd9f4445500b87ca99d70c3d38221dba1e592 WatchSource:0}: Error finding container f9b0e125dd557a8af827c2b8fc7bd9f4445500b87ca99d70c3d38221dba1e592: Status 404 returned error can't find the container with id f9b0e125dd557a8af827c2b8fc7bd9f4445500b87ca99d70c3d38221dba1e592 Nov 04 01:23:10.340279 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:10.332650 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:23:10.340279 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:10.332713 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" podUID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 04 01:23:10.447015 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:10.446979 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" event={"ID":"cb6e5ba9-6335-4740-b5d8-1c371fca6d97","Type":"ContainerStarted","Data":"f9b0e125dd557a8af827c2b8fc7bd9f4445500b87ca99d70c3d38221dba1e592"} Nov 04 01:23:10.613297 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:10.613258 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:23:10.613947 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:10.613911 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" podUID="061cb628-1307-41db-b238-e432a72f9150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 04 01:23:10.652586 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:10.652545 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:23:10.652907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:10.652879 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 04 01:23:11.332833 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:11.332793 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:23:11.333132 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:11.333085 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" podUID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 04 01:23:11.614155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:11.611742 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:23:11.614155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:11.611806 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" podUID="061cb628-1307-41db-b238-e432a72f9150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 04 01:23:11.655140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:11.654868 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:23:11.655140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:11.654943 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 04 01:23:12.331911 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:12.331870 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:23:12.332109 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:12.331941 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" podUID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 04 01:23:12.521466 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:12.521424 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" event={"ID":"cb6e5ba9-6335-4740-b5d8-1c371fca6d97","Type":"ContainerStarted","Data":"19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd"} Nov 04 01:23:12.521681 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:12.521660 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:12.608206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:12.608173 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 04 01:23:12.608404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:12.608223 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" podUID="061cb628-1307-41db-b238-e432a72f9150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 04 01:23:12.642157 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:12.640946 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" podStartSLOduration=3.440990591 podStartE2EDuration="4.640909851s" totalImagesPullingTime="1.19991926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:12.63762019 +0000 UTC m=+1538.253965950" watchObservedRunningTime="2026-11-04 01:23:12.640909851 +0000 UTC m=+1538.257255617" Nov 04 01:23:12.642703 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:12.642579 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:23:12.642703 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:12.642632 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 04 01:23:13.331953 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:13.331902 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:23:13.332197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:13.331971 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" podUID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 04 01:23:14.280553 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:14.280196 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:14.282495 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:14.282464 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:14.283643 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:14.283615 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:23:14.291984 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:14.291954 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:14.293797 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:14.292609 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:14.294609 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:14.294586 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:23:14.332285 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:14.332254 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:23:14.332451 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:14.332306 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" podUID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 04 01:23:15.138143 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:23:15.138000 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:23:15.783302 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:15.783187 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:15.784021 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:15.783918 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:15.786465 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:15.786439 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:23:18.508865 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.508831 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77"] Nov 04 01:23:18.513493 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.513469 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:18.541885 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.541855 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:23:18.561708 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.561676 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77"] Nov 04 01:23:18.563778 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.563753 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b45d1fe0-serving-cert\"" Nov 04 01:23:18.657411 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.657374 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbbg\" (UniqueName: \"kubernetes.io/projected/5ff775bf-a028-45d3-815c-7578b3235599-kube-api-access-cnbbg\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:18.657612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.657427 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff775bf-a028-45d3-815c-7578b3235599-openshift-service-ca\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:18.657685 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.657617 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-maas-api-tls\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:18.657950 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.657926 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-metrics-certs\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:18.759533 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.759447 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-maas-api-tls\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:18.759738 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:23:18.759624 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-serving-cert: secret "maas-api-e2e-migrate-b45d1fe0-serving-cert" not found Nov 04 01:23:18.759738 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.759679 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-metrics-certs\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:18.759738 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:23:18.759716 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-maas-api-tls podName:5ff775bf-a028-45d3-815c-7578b3235599 nodeName:}" failed. No retries permitted until 2026-11-04 01:23:19.259693151 +0000 UTC m=+1544.876038912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-maas-api-tls") pod "maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" (UID: "5ff775bf-a028-45d3-815c-7578b3235599") : secret "maas-api-e2e-migrate-b45d1fe0-serving-cert" not found Nov 04 01:23:18.759932 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.759851 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbbg\" (UniqueName: \"kubernetes.io/projected/5ff775bf-a028-45d3-815c-7578b3235599-kube-api-access-cnbbg\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:18.759932 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.759893 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff775bf-a028-45d3-815c-7578b3235599-openshift-service-ca\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:18.761248 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.760746 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff775bf-a028-45d3-815c-7578b3235599-openshift-service-ca\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:18.764994 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.764961 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-metrics-certs\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:18.818923 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:18.818884 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbbg\" (UniqueName: \"kubernetes.io/projected/5ff775bf-a028-45d3-815c-7578b3235599-kube-api-access-cnbbg\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:19.269747 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:19.269707 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-maas-api-tls\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:19.272356 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:19.272327 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-maas-api-tls\") pod \"maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:19.429388 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:19.429353 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:19.812585 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:19.812552 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77"] Nov 04 01:23:19.816237 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:23:19.816205 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff775bf_a028_45d3_815c_7578b3235599.slice/crio-58ab8c023ea40ecca1617a1296a278fd5e6233ca57e3090171f498e8a444a968 WatchSource:0}: Error finding container 58ab8c023ea40ecca1617a1296a278fd5e6233ca57e3090171f498e8a444a968: Status 404 returned error can't find the container with id 58ab8c023ea40ecca1617a1296a278fd5e6233ca57e3090171f498e8a444a968 Nov 04 01:23:20.625384 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:20.625346 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" event={"ID":"5ff775bf-a028-45d3-815c-7578b3235599","Type":"ContainerStarted","Data":"58ab8c023ea40ecca1617a1296a278fd5e6233ca57e3090171f498e8a444a968"} Nov 04 01:23:21.684154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:21.679073 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" event={"ID":"5ff775bf-a028-45d3-815c-7578b3235599","Type":"ContainerStarted","Data":"bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6"} Nov 04 01:23:21.684154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:21.680377 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:21.813151 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:21.804474 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" podStartSLOduration=2.83540322 podStartE2EDuration="3.804449684s" totalImagesPullingTime="969.046464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:21.798161025 +0000 UTC m=+1547.414506789" watchObservedRunningTime="2026-11-04 01:23:21.804449684 +0000 UTC m=+1547.420795451" Nov 04 01:23:28.758158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:28.756322 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:23:33.071142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:33.066749 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:23:42.433536 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.433497 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd"] Nov 04 01:23:42.439612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.439581 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.503853 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.503810 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2502a53-973b-4987-9890-f1172bf9b84d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.503853 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.503851 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.504096 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.503876 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.504096 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.504029 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.504096 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.504086 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.504359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.504155 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.504359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.504204 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2502a53-973b-4987-9890-f1172bf9b84d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.504359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.504304 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtn4\" (UniqueName: \"kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-kube-api-access-7rtn4\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.504359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.504354 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.535717 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.535683 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd"] Nov 04 01:23:42.606363 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.606318 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.606560 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.606418 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.606560 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.606450 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.606560 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.606483 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2502a53-973b-4987-9890-f1172bf9b84d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.606912 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.606882 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.607046 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.606938 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtn4\" (UniqueName: \"kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-kube-api-access-7rtn4\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.607148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.607040 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.607227 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.607194 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2502a53-973b-4987-9890-f1172bf9b84d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.607292 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.607229 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.607292 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.607234 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2502a53-973b-4987-9890-f1172bf9b84d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.607292 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.607267 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.607501 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.607476 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.607687 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.607658 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.607858 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.607833 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.609314 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.609289 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.609591 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.609573 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2502a53-973b-4987-9890-f1172bf9b84d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.662255 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.662224 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.663196 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.663164 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtn4\" (UniqueName: \"kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-kube-api-access-7rtn4\") pod \"maas-default-gateway-openshift-default-694db55c87-9d8xd\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:42.754913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:42.754833 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:43.196568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:43.196527 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd"] Nov 04 01:23:43.237420 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:23:43.237377 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2502a53_973b_4987_9890_f1172bf9b84d.slice/crio-730d93406915ed51299a7260cd6d54f3e86a96c6661e1eab8c40d247810ba810 WatchSource:0}: Error finding container 730d93406915ed51299a7260cd6d54f3e86a96c6661e1eab8c40d247810ba810: Status 404 returned error can't find the container with id 730d93406915ed51299a7260cd6d54f3e86a96c6661e1eab8c40d247810ba810 Nov 04 01:23:43.240191 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:43.240156 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:43.240289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:43.240228 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:43.240289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:43.240256 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:44.009613 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:44.009507 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" event={"ID":"f2502a53-973b-4987-9890-f1172bf9b84d","Type":"ContainerStarted","Data":"c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b"} Nov 04 01:23:44.009613 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:44.009570 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" event={"ID":"f2502a53-973b-4987-9890-f1172bf9b84d","Type":"ContainerStarted","Data":"730d93406915ed51299a7260cd6d54f3e86a96c6661e1eab8c40d247810ba810"} Nov 04 01:23:44.011331 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:44.011306 5786 generic.go:356] "Generic (PLEG): container finished" podID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerID="a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4" exitCode=0 Nov 04 01:23:44.011439 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:44.011370 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" event={"ID":"f3031f39-b918-4fa7-bfc4-b23a14bb83d1","Type":"ContainerDied","Data":"a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4"} Nov 04 01:23:44.011767 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:44.011753 5786 scope.go:122] "RemoveContainer" containerID="a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4" Nov 04 01:23:44.013238 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:44.013210 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:44.013304 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:44.013276 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:44.013352 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:44.013321 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:44.218776 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:44.218720 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" podStartSLOduration=2.218700819 podStartE2EDuration="2.218700819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:44.172598828 +0000 UTC m=+1569.788944625" watchObservedRunningTime="2026-11-04 01:23:44.218700819 +0000 UTC m=+1569.835046583" Nov 04 01:23:44.847144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:44.794161 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:44.856306 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:44.817833 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9d8xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:23:44.856632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:44.856595 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:23:45.025339 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:45.025293 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" event={"ID":"f3031f39-b918-4fa7-bfc4-b23a14bb83d1","Type":"ContainerStarted","Data":"cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985"} Nov 04 01:23:45.128531 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:23:45.128432 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:23:45.755861 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:45.755823 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9d8xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:23:45.756048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:45.755883 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:23:45.758934 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:45.758916 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:23:45.760083 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:45.760060 5786 patch_prober.go:29] interesting pod/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:23:45.760204 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:45.760098 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 01:23:46.756101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:46.756071 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9d8xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:23:46.756643 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:46.756613 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:23:46.759422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:46.759400 5786 patch_prober.go:29] interesting pod/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:23:46.759514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:46.759446 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 01:23:47.747818 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:47.741614 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:23:47.755415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:47.755388 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9d8xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:23:47.755567 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:47.755438 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:23:47.759749 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:47.759726 5786 patch_prober.go:29] interesting pod/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:23:47.760085 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:47.759763 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 01:23:48.755999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:48.755962 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9d8xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:23:48.756212 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:48.756016 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:23:48.759290 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:48.759269 5786 patch_prober.go:29] interesting pod/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:23:48.759343 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:48.759311 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 01:23:49.755662 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:49.755626 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9d8xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:23:49.756178 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:49.755689 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:23:49.759947 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:49.759920 5786 patch_prober.go:29] interesting pod/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:23:49.760079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:49.759964 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 01:23:50.755562 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:50.755524 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9d8xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:23:50.755764 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:50.755583 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:23:50.763922 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:50.763893 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:23:50.765519 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:50.765487 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:23:52.334532 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:52.334490 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:52.335739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:52.335711 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:23:52.336599 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:23:52.336579 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:24:15.269502 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:24:15.269461 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:24:16.716751 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:16.716710 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:24:27.869225 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:27.869192 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5"] Nov 04 01:24:27.993273 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:27.993237 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg"] Nov 04 01:24:32.205327 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:32.205296 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k"] Nov 04 01:24:32.441101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:32.441065 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj"] Nov 04 01:24:32.869795 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:32.869723 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" podUID="f530be7d-43e4-468d-8078-f208203d0bd0" containerName="payload-processing" containerID="cri-o://1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846" gracePeriod=25 Nov 04 01:24:33.012626 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.012571 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" podUID="8c3db980-b5ed-4433-9644-43e066dcc7e7" containerName="payload-pre-processing" containerID="cri-o://8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930" gracePeriod=25 Nov 04 01:24:33.225779 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.225068 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" Nov 04 01:24:33.268709 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.268682 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f530be7d-43e4-468d-8078-f208203d0bd0-kube-api-access-sx8rj\" (UniqueName: \"kubernetes.io/projected/f530be7d-43e4-468d-8078-f208203d0bd0-kube-api-access-sx8rj\") pod \"f530be7d-43e4-468d-8078-f208203d0bd0\" (UID: \"f530be7d-43e4-468d-8078-f208203d0bd0\") " Nov 04 01:24:33.268925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.268908 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f530be7d-43e4-468d-8078-f208203d0bd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f530be7d-43e4-468d-8078-f208203d0bd0-plugins-config-volume\") pod \"f530be7d-43e4-468d-8078-f208203d0bd0\" (UID: \"f530be7d-43e4-468d-8078-f208203d0bd0\") " Nov 04 01:24:33.269398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.269357 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f530be7d-43e4-468d-8078-f208203d0bd0-plugins-config-volume" pod "f530be7d-43e4-468d-8078-f208203d0bd0" (UID: "f530be7d-43e4-468d-8078-f208203d0bd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:33.269612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.269587 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f530be7d-43e4-468d-8078-f208203d0bd0-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:33.271005 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.270974 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f530be7d-43e4-468d-8078-f208203d0bd0-kube-api-access-sx8rj" pod "f530be7d-43e4-468d-8078-f208203d0bd0" (UID: "f530be7d-43e4-468d-8078-f208203d0bd0"). InnerVolumeSpecName "kube-api-access-sx8rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:33.293364 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.293343 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" Nov 04 01:24:33.371086 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.371055 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3db980-b5ed-4433-9644-43e066dcc7e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3db980-b5ed-4433-9644-43e066dcc7e7-plugins-config-volume\") pod \"8c3db980-b5ed-4433-9644-43e066dcc7e7\" (UID: \"8c3db980-b5ed-4433-9644-43e066dcc7e7\") " Nov 04 01:24:33.371432 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.371407 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3db980-b5ed-4433-9644-43e066dcc7e7-plugins-config-volume" pod "8c3db980-b5ed-4433-9644-43e066dcc7e7" (UID: "8c3db980-b5ed-4433-9644-43e066dcc7e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:33.371532 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.371426 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3db980-b5ed-4433-9644-43e066dcc7e7-kube-api-access-p4m8w\" (UniqueName: \"kubernetes.io/projected/8c3db980-b5ed-4433-9644-43e066dcc7e7-kube-api-access-p4m8w\") pod \"8c3db980-b5ed-4433-9644-43e066dcc7e7\" (UID: \"8c3db980-b5ed-4433-9644-43e066dcc7e7\") " Nov 04 01:24:33.371942 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.371923 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3db980-b5ed-4433-9644-43e066dcc7e7-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:33.372016 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.371946 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8rj\" (UniqueName: \"kubernetes.io/projected/f530be7d-43e4-468d-8078-f208203d0bd0-kube-api-access-sx8rj\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:33.373431 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.373411 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3db980-b5ed-4433-9644-43e066dcc7e7-kube-api-access-p4m8w" pod "8c3db980-b5ed-4433-9644-43e066dcc7e7" (UID: "8c3db980-b5ed-4433-9644-43e066dcc7e7"). InnerVolumeSpecName "kube-api-access-p4m8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:33.473853 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.473779 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4m8w\" (UniqueName: \"kubernetes.io/projected/8c3db980-b5ed-4433-9644-43e066dcc7e7-kube-api-access-p4m8w\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:33.990341 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.990304 5786 generic.go:356] "Generic (PLEG): container finished" podID="f530be7d-43e4-468d-8078-f208203d0bd0" containerID="1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846" exitCode=0 Nov 04 01:24:33.990675 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.990649 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" event={"ID":"f530be7d-43e4-468d-8078-f208203d0bd0","Type":"ContainerDied","Data":"1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846"} Nov 04 01:24:33.990852 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.990836 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" event={"ID":"f530be7d-43e4-468d-8078-f208203d0bd0","Type":"ContainerDied","Data":"7e081da76f14264bc0cce05d86f41fdfbaa2f76e0f79fae9a25e50a9b3156890"} Nov 04 01:24:33.990967 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.990953 5786 scope.go:122] "RemoveContainer" containerID="1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846" Nov 04 01:24:33.991237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:33.991219 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5" Nov 04 01:24:34.068171 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.058448 5786 generic.go:356] "Generic (PLEG): container finished" podID="8c3db980-b5ed-4433-9644-43e066dcc7e7" containerID="8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930" exitCode=0 Nov 04 01:24:34.068171 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.058697 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" event={"ID":"8c3db980-b5ed-4433-9644-43e066dcc7e7","Type":"ContainerDied","Data":"8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930"} Nov 04 01:24:34.068171 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.058761 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" event={"ID":"8c3db980-b5ed-4433-9644-43e066dcc7e7","Type":"ContainerDied","Data":"b5e129781913da71cf7bf38305c60ebb8652f15aa4a9c965f13f0a6755bcb2a0"} Nov 04 01:24:34.068171 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.058853 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg" Nov 04 01:24:34.182027 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.180967 5786 scope.go:122] "RemoveContainer" containerID="1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846" Nov 04 01:24:34.182027 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:24:34.181845 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846\": container with ID starting with 1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846 not found: ID does not exist" containerID="1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846" Nov 04 01:24:34.182027 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.181891 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846"} err="failed to get container status \"1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846\": rpc error: code = NotFound desc = could not find container \"1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846\": container with ID starting with 1eaec5d8b33b8a7bf421e8f9e2bb91bf83864423dea53a4012c32dcb3742a846 not found: ID does not exist" Nov 04 01:24:34.182027 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.181913 5786 scope.go:122] "RemoveContainer" containerID="8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930" Nov 04 01:24:34.199796 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.199312 5786 scope.go:122] "RemoveContainer" containerID="8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930" Nov 04 01:24:34.200514 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:24:34.200052 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930\": container with ID starting with 8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930 not found: ID does not exist" containerID="8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930" Nov 04 01:24:34.200514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.200092 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930"} err="failed to get container status \"8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930\": rpc error: code = NotFound desc = could not find container \"8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930\": container with ID starting with 8ea0e7d44918bf77bb45504314a1e3ab20b062b8cfefafd85346e54262ac3930 not found: ID does not exist" Nov 04 01:24:34.269472 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.269221 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5"] Nov 04 01:24:34.305140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.299607 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-60398d45-8df775678-skcn5"] Nov 04 01:24:34.421563 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.421309 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg"] Nov 04 01:24:34.433554 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.433471 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-60398d45-84df4c9b75-rcxxg"] Nov 04 01:24:34.605097 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.605038 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3db980-b5ed-4433-9644-43e066dcc7e7" path="/var/lib/kubelet/pods/8c3db980-b5ed-4433-9644-43e066dcc7e7/volumes" Nov 04 01:24:34.612059 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:34.612031 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f530be7d-43e4-468d-8078-f208203d0bd0" path="/var/lib/kubelet/pods/f530be7d-43e4-468d-8078-f208203d0bd0/volumes" Nov 04 01:24:37.206071 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.206031 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" podUID="cc52bbbc-616f-4a2a-be0e-33d9294159bf" containerName="payload-processing" containerID="cri-o://3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c" gracePeriod=25 Nov 04 01:24:37.441861 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.441821 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" podUID="9f373a64-18f6-4b78-ba0c-dc506a788e27" containerName="payload-pre-processing" containerID="cri-o://01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e" gracePeriod=25 Nov 04 01:24:37.579203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.579178 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" Nov 04 01:24:37.600496 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.600466 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc52bbbc-616f-4a2a-be0e-33d9294159bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc52bbbc-616f-4a2a-be0e-33d9294159bf-plugins-config-volume\") pod \"cc52bbbc-616f-4a2a-be0e-33d9294159bf\" (UID: \"cc52bbbc-616f-4a2a-be0e-33d9294159bf\") " Nov 04 01:24:37.600770 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.600750 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc52bbbc-616f-4a2a-be0e-33d9294159bf-kube-api-access-q55tj\" (UniqueName: \"kubernetes.io/projected/cc52bbbc-616f-4a2a-be0e-33d9294159bf-kube-api-access-q55tj\") pod \"cc52bbbc-616f-4a2a-be0e-33d9294159bf\" (UID: \"cc52bbbc-616f-4a2a-be0e-33d9294159bf\") " Nov 04 01:24:37.600887 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.600859 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc52bbbc-616f-4a2a-be0e-33d9294159bf-plugins-config-volume" pod "cc52bbbc-616f-4a2a-be0e-33d9294159bf" (UID: "cc52bbbc-616f-4a2a-be0e-33d9294159bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:37.601532 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.601499 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc52bbbc-616f-4a2a-be0e-33d9294159bf-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:37.602968 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.602947 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc52bbbc-616f-4a2a-be0e-33d9294159bf-kube-api-access-q55tj" pod "cc52bbbc-616f-4a2a-be0e-33d9294159bf" (UID: "cc52bbbc-616f-4a2a-be0e-33d9294159bf"). InnerVolumeSpecName "kube-api-access-q55tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:37.699186 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.698904 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" Nov 04 01:24:37.703644 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.703611 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q55tj\" (UniqueName: \"kubernetes.io/projected/cc52bbbc-616f-4a2a-be0e-33d9294159bf-kube-api-access-q55tj\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:37.805135 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.805042 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f373a64-18f6-4b78-ba0c-dc506a788e27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f373a64-18f6-4b78-ba0c-dc506a788e27-plugins-config-volume\") pod \"9f373a64-18f6-4b78-ba0c-dc506a788e27\" (UID: \"9f373a64-18f6-4b78-ba0c-dc506a788e27\") " Nov 04 01:24:37.805395 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.805374 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f373a64-18f6-4b78-ba0c-dc506a788e27-kube-api-access-p7kps\" (UniqueName: \"kubernetes.io/projected/9f373a64-18f6-4b78-ba0c-dc506a788e27-kube-api-access-p7kps\") pod \"9f373a64-18f6-4b78-ba0c-dc506a788e27\" (UID: \"9f373a64-18f6-4b78-ba0c-dc506a788e27\") " Nov 04 01:24:37.805519 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.805499 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f373a64-18f6-4b78-ba0c-dc506a788e27-plugins-config-volume" pod "9f373a64-18f6-4b78-ba0c-dc506a788e27" (UID: "9f373a64-18f6-4b78-ba0c-dc506a788e27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:37.805892 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.805869 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f373a64-18f6-4b78-ba0c-dc506a788e27-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:37.807421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:37.807400 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f373a64-18f6-4b78-ba0c-dc506a788e27-kube-api-access-p7kps" pod "9f373a64-18f6-4b78-ba0c-dc506a788e27" (UID: "9f373a64-18f6-4b78-ba0c-dc506a788e27"). InnerVolumeSpecName "kube-api-access-p7kps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:38.041142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.040200 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7kps\" (UniqueName: \"kubernetes.io/projected/9f373a64-18f6-4b78-ba0c-dc506a788e27-kube-api-access-p7kps\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:38.197156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.194983 5786 generic.go:356] "Generic (PLEG): container finished" podID="cc52bbbc-616f-4a2a-be0e-33d9294159bf" containerID="3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c" exitCode=0 Nov 04 01:24:38.197156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.195340 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" event={"ID":"cc52bbbc-616f-4a2a-be0e-33d9294159bf","Type":"ContainerDied","Data":"3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c"} Nov 04 01:24:38.197156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.195400 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" event={"ID":"cc52bbbc-616f-4a2a-be0e-33d9294159bf","Type":"ContainerDied","Data":"6fcb50a436c5d11452278e2ebb01bef4b22bc2a07e1f87755be1784ee89854b5"} Nov 04 01:24:38.197156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.195431 5786 scope.go:122] "RemoveContainer" containerID="3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c" Nov 04 01:24:38.197156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.195589 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k" Nov 04 01:24:38.244183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.224723 5786 generic.go:356] "Generic (PLEG): container finished" podID="9f373a64-18f6-4b78-ba0c-dc506a788e27" containerID="01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e" exitCode=0 Nov 04 01:24:38.244183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.224776 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" event={"ID":"9f373a64-18f6-4b78-ba0c-dc506a788e27","Type":"ContainerDied","Data":"01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e"} Nov 04 01:24:38.244183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.224825 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" event={"ID":"9f373a64-18f6-4b78-ba0c-dc506a788e27","Type":"ContainerDied","Data":"681492429f2d9b6a0b9652ce3b539dcdb62d6c1fe8fcda3a0529b7674d434b27"} Nov 04 01:24:38.244183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.224897 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj" Nov 04 01:24:38.277278 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.277095 5786 scope.go:122] "RemoveContainer" containerID="3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c" Nov 04 01:24:38.277633 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:24:38.277546 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c\": container with ID starting with 3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c not found: ID does not exist" containerID="3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c" Nov 04 01:24:38.277633 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.277583 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c"} err="failed to get container status \"3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c\": rpc error: code = NotFound desc = could not find container \"3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c\": container with ID starting with 3a8fad7011ae4d2b3cfa9c351eb8ab253ed7216b6c1bbba41bb2183628ef3b9c not found: ID does not exist" Nov 04 01:24:38.277633 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.277601 5786 scope.go:122] "RemoveContainer" containerID="01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e" Nov 04 01:24:38.303312 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.299707 5786 scope.go:122] "RemoveContainer" containerID="01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e" Nov 04 01:24:38.303312 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:24:38.300333 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e\": container with ID starting with 01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e not found: ID does not exist" containerID="01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e" Nov 04 01:24:38.303312 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.300373 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e"} err="failed to get container status \"01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e\": rpc error: code = NotFound desc = could not find container \"01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e\": container with ID starting with 01934322c7889f568f28b88f0f687fbb82197314a502170b98efe736d8dfd01e not found: ID does not exist" Nov 04 01:24:38.462186 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.458182 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj"] Nov 04 01:24:38.478142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.474436 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e400a3a6-5d5576fb79-gz5bj"] Nov 04 01:24:38.537076 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.537038 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f373a64-18f6-4b78-ba0c-dc506a788e27" path="/var/lib/kubelet/pods/9f373a64-18f6-4b78-ba0c-dc506a788e27/volumes" Nov 04 01:24:38.568702 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.568667 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k"] Nov 04 01:24:38.574503 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:38.574464 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e400a3a6-64449c7d46-sx55k"] Nov 04 01:24:40.507391 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:24:40.507359 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc52bbbc-616f-4a2a-be0e-33d9294159bf" path="/var/lib/kubelet/pods/cc52bbbc-616f-4a2a-be0e-33d9294159bf/volumes" Nov 04 01:24:45.088440 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:24:45.088353 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:25:02.308949 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.308599 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv"] Nov 04 01:25:02.312147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.309513 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" podUID="cb6e5ba9-6335-4740-b5d8-1c371fca6d97" containerName="maas-api" containerID="cri-o://19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd" gracePeriod=30 Nov 04 01:25:02.744334 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.744303 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:25:02.765902 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.765872 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-maas-api-tls\") pod \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " Nov 04 01:25:02.766055 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.765913 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-metrics-certs\") pod \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " Nov 04 01:25:02.766055 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.766015 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-kube-api-access-4phsw\" (UniqueName: \"kubernetes.io/projected/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-kube-api-access-4phsw\") pod \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " Nov 04 01:25:02.766285 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.766193 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-openshift-service-ca\") pod \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\" (UID: \"cb6e5ba9-6335-4740-b5d8-1c371fca6d97\") " Nov 04 01:25:02.766592 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.766564 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-openshift-service-ca" pod "cb6e5ba9-6335-4740-b5d8-1c371fca6d97" (UID: "cb6e5ba9-6335-4740-b5d8-1c371fca6d97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:02.766843 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.766826 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:02.768224 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.768203 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-kube-api-access-4phsw" pod "cb6e5ba9-6335-4740-b5d8-1c371fca6d97" (UID: "cb6e5ba9-6335-4740-b5d8-1c371fca6d97"). InnerVolumeSpecName "kube-api-access-4phsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:02.768561 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.768539 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-metrics-certs" pod "cb6e5ba9-6335-4740-b5d8-1c371fca6d97" (UID: "cb6e5ba9-6335-4740-b5d8-1c371fca6d97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:02.768561 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.768554 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-maas-api-tls" pod "cb6e5ba9-6335-4740-b5d8-1c371fca6d97" (UID: "cb6e5ba9-6335-4740-b5d8-1c371fca6d97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:02.968226 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.884415 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:02.968226 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.884450 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:02.968226 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:02.884466 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4phsw\" (UniqueName: \"kubernetes.io/projected/cb6e5ba9-6335-4740-b5d8-1c371fca6d97-kube-api-access-4phsw\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:03.449757 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:03.449712 5786 generic.go:356] "Generic (PLEG): container finished" podID="cb6e5ba9-6335-4740-b5d8-1c371fca6d97" containerID="19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd" exitCode=0 Nov 04 01:25:03.450578 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:03.450539 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" event={"ID":"cb6e5ba9-6335-4740-b5d8-1c371fca6d97","Type":"ContainerDied","Data":"19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd"} Nov 04 01:25:03.450771 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:03.450748 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" event={"ID":"cb6e5ba9-6335-4740-b5d8-1c371fca6d97","Type":"ContainerDied","Data":"f9b0e125dd557a8af827c2b8fc7bd9f4445500b87ca99d70c3d38221dba1e592"} Nov 04 01:25:03.450893 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:03.450881 5786 scope.go:122] "RemoveContainer" containerID="19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd" Nov 04 01:25:03.451190 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:03.451168 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv" Nov 04 01:25:03.603173 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:03.602624 5786 scope.go:122] "RemoveContainer" containerID="19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd" Nov 04 01:25:03.603353 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:25:03.603182 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd\": container with ID starting with 19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd not found: ID does not exist" containerID="19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd" Nov 04 01:25:03.603353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:03.603214 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd"} err="failed to get container status \"19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd\": rpc error: code = NotFound desc = could not find container \"19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd\": container with ID starting with 19b6578d065cc016e689f5f6d0ce7303b6024c48bc965979f5ad6f53eb5ea7fd not found: ID does not exist" Nov 04 01:25:03.624923 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:03.624262 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv"] Nov 04 01:25:03.639846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:03.639802 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-60398d45-64776c497c-sqfvv"] Nov 04 01:25:04.559439 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:04.559405 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6e5ba9-6335-4740-b5d8-1c371fca6d97" path="/var/lib/kubelet/pods/cb6e5ba9-6335-4740-b5d8-1c371fca6d97/volumes" Nov 04 01:25:04.960832 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:04.960802 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz"] Nov 04 01:25:04.961666 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:04.961636 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" containerID="cri-o://cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985" gracePeriod=30 Nov 04 01:25:04.964681 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:04.964659 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77"] Nov 04 01:25:04.964852 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:04.964831 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" podUID="5ff775bf-a028-45d3-815c-7578b3235599" containerName="maas-api" containerID="cri-o://bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6" gracePeriod=30 Nov 04 01:25:05.049101 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:25:05.049069 5786 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff775bf_a028_45d3_815c_7578b3235599.slice/crio-conmon-bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:25:05.609148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.597632 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:25:05.625958 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.622403 5786 generic.go:356] "Generic (PLEG): container finished" podID="5ff775bf-a028-45d3-815c-7578b3235599" containerID="bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6" exitCode=0 Nov 04 01:25:05.625958 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.622730 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" event={"ID":"5ff775bf-a028-45d3-815c-7578b3235599","Type":"ContainerDied","Data":"bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6"} Nov 04 01:25:05.625958 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.622800 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" event={"ID":"5ff775bf-a028-45d3-815c-7578b3235599","Type":"ContainerDied","Data":"58ab8c023ea40ecca1617a1296a278fd5e6233ca57e3090171f498e8a444a968"} Nov 04 01:25:05.625958 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.622843 5786 scope.go:122] "RemoveContainer" containerID="bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6" Nov 04 01:25:05.625958 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.622905 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77" Nov 04 01:25:05.762582 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.760105 5786 scope.go:122] "RemoveContainer" containerID="bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6" Nov 04 01:25:05.762582 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.760802 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-maas-api-tls\") pod \"5ff775bf-a028-45d3-815c-7578b3235599\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " Nov 04 01:25:05.762582 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.760869 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-metrics-certs\") pod \"5ff775bf-a028-45d3-815c-7578b3235599\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " Nov 04 01:25:05.762582 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.760968 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff775bf-a028-45d3-815c-7578b3235599-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff775bf-a028-45d3-815c-7578b3235599-openshift-service-ca\") pod \"5ff775bf-a028-45d3-815c-7578b3235599\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " Nov 04 01:25:05.762582 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.761037 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff775bf-a028-45d3-815c-7578b3235599-kube-api-access-cnbbg\" (UniqueName: \"kubernetes.io/projected/5ff775bf-a028-45d3-815c-7578b3235599-kube-api-access-cnbbg\") pod \"5ff775bf-a028-45d3-815c-7578b3235599\" (UID: \"5ff775bf-a028-45d3-815c-7578b3235599\") " Nov 04 01:25:05.763933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.763665 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff775bf-a028-45d3-815c-7578b3235599-openshift-service-ca" pod "5ff775bf-a028-45d3-815c-7578b3235599" (UID: "5ff775bf-a028-45d3-815c-7578b3235599"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:05.766033 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:25:05.765941 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6\": container with ID starting with bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6 not found: ID does not exist" containerID="bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6" Nov 04 01:25:05.766033 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.766002 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6"} err="failed to get container status \"bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6\": rpc error: code = NotFound desc = could not find container \"bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6\": container with ID starting with bb8866c5fbeea1e3672ab30f6bcc615e48885307e93b321c19e47cf80e21acb6 not found: ID does not exist" Nov 04 01:25:05.779760 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.779728 5786 patch_prober.go:29] interesting pod/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:25:05.780057 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.780032 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:25:05.793429 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.792279 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff775bf-a028-45d3-815c-7578b3235599-kube-api-access-cnbbg" pod "5ff775bf-a028-45d3-815c-7578b3235599" (UID: "5ff775bf-a028-45d3-815c-7578b3235599"). InnerVolumeSpecName "kube-api-access-cnbbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:05.793429 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.792394 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-metrics-certs" pod "5ff775bf-a028-45d3-815c-7578b3235599" (UID: "5ff775bf-a028-45d3-815c-7578b3235599"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:05.803632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.798226 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-maas-api-tls" pod "5ff775bf-a028-45d3-815c-7578b3235599" (UID: "5ff775bf-a028-45d3-815c-7578b3235599"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:05.863509 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.863475 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:05.863509 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.863511 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff775bf-a028-45d3-815c-7578b3235599-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:05.863752 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.863526 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff775bf-a028-45d3-815c-7578b3235599-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:05.863752 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:05.863541 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnbbg\" (UniqueName: \"kubernetes.io/projected/5ff775bf-a028-45d3-815c-7578b3235599-kube-api-access-cnbbg\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:06.083240 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:06.079922 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77"] Nov 04 01:25:06.106142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:06.105199 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b45d1fe0-8b4fd9447-w8s77"] Nov 04 01:25:06.545143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:06.544579 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff775bf-a028-45d3-815c-7578b3235599" path="/var/lib/kubelet/pods/5ff775bf-a028-45d3-815c-7578b3235599/volumes" Nov 04 01:25:08.575053 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575019 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j"] Nov 04 01:25:08.575748 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575731 5786 cpu_manager.go:414] "containerMap: removing container" podUID="8c3db980-b5ed-4433-9644-43e066dcc7e7" containerName="payload-pre-processing" Nov 04 01:25:08.575815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575749 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3db980-b5ed-4433-9644-43e066dcc7e7" containerName="payload-pre-processing" Nov 04 01:25:08.575815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575780 5786 cpu_manager.go:414] "containerMap: removing container" podUID="f530be7d-43e4-468d-8078-f208203d0bd0" containerName="payload-processing" Nov 04 01:25:08.575815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575786 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f530be7d-43e4-468d-8078-f208203d0bd0" containerName="payload-processing" Nov 04 01:25:08.575815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575798 5786 cpu_manager.go:414] "containerMap: removing container" podUID="cc52bbbc-616f-4a2a-be0e-33d9294159bf" containerName="payload-processing" Nov 04 01:25:08.575815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575803 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc52bbbc-616f-4a2a-be0e-33d9294159bf" containerName="payload-processing" Nov 04 01:25:08.575815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575811 5786 cpu_manager.go:414] "containerMap: removing container" podUID="cb6e5ba9-6335-4740-b5d8-1c371fca6d97" containerName="maas-api" Nov 04 01:25:08.576000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575818 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6e5ba9-6335-4740-b5d8-1c371fca6d97" containerName="maas-api" Nov 04 01:25:08.576000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575857 5786 cpu_manager.go:414] "containerMap: removing container" podUID="9f373a64-18f6-4b78-ba0c-dc506a788e27" containerName="payload-pre-processing" Nov 04 01:25:08.576000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575865 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f373a64-18f6-4b78-ba0c-dc506a788e27" containerName="payload-pre-processing" Nov 04 01:25:08.576000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575880 5786 cpu_manager.go:414] "containerMap: removing container" podUID="5ff775bf-a028-45d3-815c-7578b3235599" containerName="maas-api" Nov 04 01:25:08.576000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.575887 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff775bf-a028-45d3-815c-7578b3235599" containerName="maas-api" Nov 04 01:25:08.576168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.576030 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3db980-b5ed-4433-9644-43e066dcc7e7" containerName="payload-pre-processing" Nov 04 01:25:08.576168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.576040 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="f530be7d-43e4-468d-8078-f208203d0bd0" containerName="payload-processing" Nov 04 01:25:08.576168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.576048 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc52bbbc-616f-4a2a-be0e-33d9294159bf" containerName="payload-processing" Nov 04 01:25:08.576168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.576053 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6e5ba9-6335-4740-b5d8-1c371fca6d97" containerName="maas-api" Nov 04 01:25:08.576168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.576063 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f373a64-18f6-4b78-ba0c-dc506a788e27" containerName="payload-pre-processing" Nov 04 01:25:08.576168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.576070 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff775bf-a028-45d3-815c-7578b3235599" containerName="maas-api" Nov 04 01:25:08.581544 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.581522 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.644877 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.644849 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d3d6ea20-openshift-default-dockercfg-zzwxn\"" Nov 04 01:25:08.699832 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.699795 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j"] Nov 04 01:25:08.738729 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.738691 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnn5t\" (UniqueName: \"kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-kube-api-access-rnn5t\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.738899 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.738740 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-envoy\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.738899 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.738845 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-token\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.738989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.738948 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-podinfo\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.738989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.738975 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istiod-ca-cert\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.739064 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.739015 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-data\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.739206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.739192 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-socket\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.739251 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.739222 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-credential-socket\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.739320 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.739302 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-certs\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.840600 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.840506 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnn5t\" (UniqueName: \"kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-kube-api-access-rnn5t\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.840600 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.840560 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-envoy\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.840836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.840632 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-token\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.840836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.840716 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-podinfo\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.840836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.840741 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istiod-ca-cert\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.840836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.840799 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-data\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.841037 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.840912 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-socket\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.841037 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.840950 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-credential-socket\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.841037 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.841010 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-certs\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.841396 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.841368 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-data\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.841674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.841649 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-socket\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.841771 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.841744 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-credential-socket\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.842076 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.842054 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istiod-ca-cert\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.842223 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.842206 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-certs\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.843347 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.843321 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-envoy\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.843444 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.843395 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-podinfo\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.902777 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.902737 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnn5t\" (UniqueName: \"kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-kube-api-access-rnn5t\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:08.910225 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:08.910194 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-token\") pod \"e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:09.203142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.195814 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:09.458307 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.458216 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn"] Nov 04 01:25:09.464714 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.464686 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.525912 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.525877 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f10575b3-openshift-default-dockercfg-7rkq4\"" Nov 04 01:25:09.577544 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.577496 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn"] Nov 04 01:25:09.590342 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.590307 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1bf62f-188c-4b74-876d-da9212788972-istio-podinfo\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.590542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.590363 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-data\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.590616 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.590563 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-envoy\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.590679 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.590610 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-certs\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.590737 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.590699 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1bf62f-188c-4b74-876d-da9212788972-istiod-ca-cert\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.590795 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.590759 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-credential-socket\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.590931 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.590858 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-socket\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.591019 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.590939 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-istio-token\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.591082 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.591012 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffj2j\" (UniqueName: \"kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-kube-api-access-ffj2j\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.693953 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.693913 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1bf62f-188c-4b74-876d-da9212788972-istio-podinfo\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.694240 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.694226 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-data\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.694420 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.694408 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-envoy\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.694587 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.694574 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-certs\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.694774 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.694745 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1bf62f-188c-4b74-876d-da9212788972-istiod-ca-cert\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.694955 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.694940 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-credential-socket\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.695155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.695090 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-socket\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.695359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.695323 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-istio-token\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.695562 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.695545 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffj2j\" (UniqueName: \"kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-kube-api-access-ffj2j\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.699170 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.697704 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1bf62f-188c-4b74-876d-da9212788972-istio-podinfo\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.699170 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.698066 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-data\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.699676 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.699404 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-certs\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.700309 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.699985 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1bf62f-188c-4b74-876d-da9212788972-istiod-ca-cert\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.700309 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.700258 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-socket\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.700464 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.700314 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-envoy\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.700526 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.700483 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-credential-socket\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.759552 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.759471 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-istio-token\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.763327 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.763290 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j"] Nov 04 01:25:09.768362 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:25:09.768320 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7181de9e_3d8a_409e_bdd8_de4afd7b4be1.slice/crio-17dfcead92a79568688d299fbdb2fde4d425304592df05250e2a86a555d49093 WatchSource:0}: Error finding container 17dfcead92a79568688d299fbdb2fde4d425304592df05250e2a86a555d49093: Status 404 returned error can't find the container with id 17dfcead92a79568688d299fbdb2fde4d425304592df05250e2a86a555d49093 Nov 04 01:25:09.775332 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.773919 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:09.775332 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.774006 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:09.775332 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.774046 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:09.815472 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.812331 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffj2j\" (UniqueName: \"kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-kube-api-access-ffj2j\") pod \"e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:09.815472 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:09.812642 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:10.398819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.398749 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn"] Nov 04 01:25:10.401800 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:25:10.401772 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1bf62f_188c_4b74_876d_da9212788972.slice/crio-a0bb2eb96a742c916849ee2b57449f5ff2377abf5c1162592f3d57a4e34687cf WatchSource:0}: Error finding container a0bb2eb96a742c916849ee2b57449f5ff2377abf5c1162592f3d57a4e34687cf: Status 404 returned error can't find the container with id a0bb2eb96a742c916849ee2b57449f5ff2377abf5c1162592f3d57a4e34687cf Nov 04 01:25:10.404393 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.404361 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:10.404494 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.404425 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:10.404494 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.404462 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:10.657072 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.656991 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:25:10.731986 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.731945 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-envoy\") pod \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " Nov 04 01:25:10.731986 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.731985 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-certs\") pod \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " Nov 04 01:25:10.732282 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.732041 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-kube-api-access-fmcnf\" (UniqueName: \"kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-kube-api-access-fmcnf\") pod \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " Nov 04 01:25:10.732282 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.732082 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-data\") pod \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " Nov 04 01:25:10.732282 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.732174 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-token\" (UniqueName: \"kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-token\") pod \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " Nov 04 01:25:10.732465 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.732287 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-socket\") pod \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " Nov 04 01:25:10.732465 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.732319 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-podinfo\") pod \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " Nov 04 01:25:10.732465 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.732374 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-certs" pod "f3031f39-b918-4fa7-bfc4-b23a14bb83d1" (UID: "f3031f39-b918-4fa7-bfc4-b23a14bb83d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:10.732465 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.732383 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istiod-ca-cert\") pod \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " Nov 04 01:25:10.732679 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.732486 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-credential-socket\") pod \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\" (UID: \"f3031f39-b918-4fa7-bfc4-b23a14bb83d1\") " Nov 04 01:25:10.732679 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.732475 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-data" pod "f3031f39-b918-4fa7-bfc4-b23a14bb83d1" (UID: "f3031f39-b918-4fa7-bfc4-b23a14bb83d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:10.732787 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.732687 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istiod-ca-cert" pod "f3031f39-b918-4fa7-bfc4-b23a14bb83d1" (UID: "f3031f39-b918-4fa7-bfc4-b23a14bb83d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:10.732879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.732808 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-socket" pod "f3031f39-b918-4fa7-bfc4-b23a14bb83d1" (UID: "f3031f39-b918-4fa7-bfc4-b23a14bb83d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:10.733052 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.733025 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-credential-socket" pod "f3031f39-b918-4fa7-bfc4-b23a14bb83d1" (UID: "f3031f39-b918-4fa7-bfc4-b23a14bb83d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:10.733438 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.733413 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:10.733546 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.733442 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:10.733546 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.733460 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:10.733546 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.733475 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:10.733546 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.733492 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:10.734668 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.734645 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-token" pod "f3031f39-b918-4fa7-bfc4-b23a14bb83d1" (UID: "f3031f39-b918-4fa7-bfc4-b23a14bb83d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:10.734785 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.734728 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-podinfo" pod "f3031f39-b918-4fa7-bfc4-b23a14bb83d1" (UID: "f3031f39-b918-4fa7-bfc4-b23a14bb83d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:25:10.734785 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.734752 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-envoy" pod "f3031f39-b918-4fa7-bfc4-b23a14bb83d1" (UID: "f3031f39-b918-4fa7-bfc4-b23a14bb83d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:10.734785 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.734780 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-kube-api-access-fmcnf" pod "f3031f39-b918-4fa7-bfc4-b23a14bb83d1" (UID: "f3031f39-b918-4fa7-bfc4-b23a14bb83d1"). InnerVolumeSpecName "kube-api-access-fmcnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:10.753343 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.753307 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" event={"ID":"7181de9e-3d8a-409e-bdd8-de4afd7b4be1","Type":"ContainerStarted","Data":"7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf"} Nov 04 01:25:10.753530 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.753368 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" event={"ID":"7181de9e-3d8a-409e-bdd8-de4afd7b4be1","Type":"ContainerStarted","Data":"17dfcead92a79568688d299fbdb2fde4d425304592df05250e2a86a555d49093"} Nov 04 01:25:10.755079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.755050 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" event={"ID":"8e1bf62f-188c-4b74-876d-da9212788972","Type":"ContainerStarted","Data":"d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84"} Nov 04 01:25:10.755232 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.755104 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" event={"ID":"8e1bf62f-188c-4b74-876d-da9212788972","Type":"ContainerStarted","Data":"a0bb2eb96a742c916849ee2b57449f5ff2377abf5c1162592f3d57a4e34687cf"} Nov 04 01:25:10.756859 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.756831 5786 generic.go:356] "Generic (PLEG): container finished" podID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerID="cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985" exitCode=0 Nov 04 01:25:10.756974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.756872 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" Nov 04 01:25:10.756974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.756877 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" event={"ID":"f3031f39-b918-4fa7-bfc4-b23a14bb83d1","Type":"ContainerDied","Data":"cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985"} Nov 04 01:25:10.756974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.756919 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz" event={"ID":"f3031f39-b918-4fa7-bfc4-b23a14bb83d1","Type":"ContainerDied","Data":"29d36faa34712cbe8484c6bfb3376d4e1100453641751b373e08d783a0299f09"} Nov 04 01:25:10.756974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.756951 5786 scope.go:122] "RemoveContainer" containerID="cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985" Nov 04 01:25:10.767628 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.767610 5786 scope.go:122] "RemoveContainer" containerID="a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4" Nov 04 01:25:10.776843 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.776826 5786 scope.go:122] "RemoveContainer" containerID="cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985" Nov 04 01:25:10.777078 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:25:10.777054 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985\": container with ID starting with cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985 not found: ID does not exist" containerID="cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985" Nov 04 01:25:10.777169 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.777088 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985"} err="failed to get container status \"cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985\": rpc error: code = NotFound desc = could not find container \"cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985\": container with ID starting with cff0be783bc8305fd11966da63f4bdcf37e93ded04dfcac22482d4e0db3ba985 not found: ID does not exist" Nov 04 01:25:10.777169 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.777109 5786 scope.go:122] "RemoveContainer" containerID="a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4" Nov 04 01:25:10.777377 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:25:10.777360 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4\": container with ID starting with a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4 not found: ID does not exist" containerID="a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4" Nov 04 01:25:10.777441 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.777381 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4"} err="failed to get container status \"a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4\": rpc error: code = NotFound desc = could not find container \"a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4\": container with ID starting with a923cad425faad408be0d3c2042283d46f3648ae365725ab41a28460057e03f4 not found: ID does not exist" Nov 04 01:25:10.813607 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.813568 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:10.836521 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.835527 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:10.836521 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.835557 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:10.836521 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.835573 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmcnf\" (UniqueName: \"kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-kube-api-access-fmcnf\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:10.836521 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.835587 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3031f39-b918-4fa7-bfc4-b23a14bb83d1-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:10.890807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.890779 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz"] Nov 04 01:25:10.894062 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:10.894026 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-60398d45-openshift-default-64d6d7cd57-nqsvz"] Nov 04 01:25:11.156144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:11.148098 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" podStartSLOduration=2.148076776 podStartE2EDuration="2.148076776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:11.011834182 +0000 UTC m=+1656.628179944" watchObservedRunningTime="2026-11-04 01:25:11.148076776 +0000 UTC m=+1656.764422540" Nov 04 01:25:11.217239 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:11.206619 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:11.237170 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:11.230168 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:11.392916 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:11.392855 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" podStartSLOduration=3.392833017 podStartE2EDuration="3.392833017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:11.170974713 +0000 UTC m=+1656.787320476" watchObservedRunningTime="2026-11-04 01:25:11.392833017 +0000 UTC m=+1657.009178781" Nov 04 01:25:11.765208 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:11.765168 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:11.766207 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:11.766185 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:25:11.820529 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:11.820497 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:12.508035 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:12.507990 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" path="/var/lib/kubelet/pods/f3031f39-b918-4fa7-bfc4-b23a14bb83d1/volumes" Nov 04 01:25:12.771991 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:12.771899 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:12.772885 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:12.772867 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:25:13.138653 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.138619 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7"] Nov 04 01:25:13.139323 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.139306 5786 cpu_manager.go:414] "containerMap: removing container" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" Nov 04 01:25:13.139386 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.139324 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" Nov 04 01:25:13.139456 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.139446 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" Nov 04 01:25:13.139491 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.139466 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" Nov 04 01:25:13.148091 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.148048 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.204691 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.204662 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7"] Nov 04 01:25:13.217650 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.217615 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.217858 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.217668 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4faf119c-25e3-4f4e-80d7-998f0f46a492-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.217858 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.217773 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp4pb\" (UniqueName: \"kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-kube-api-access-vp4pb\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.217858 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.217849 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.217990 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.217935 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.218043 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.218011 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.218174 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.218156 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.218326 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.218310 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.218444 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.218428 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.320311 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.320279 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.320523 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.320353 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.320523 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.320379 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4faf119c-25e3-4f4e-80d7-998f0f46a492-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.320523 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.320396 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp4pb\" (UniqueName: \"kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-kube-api-access-vp4pb\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.320523 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.320426 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.320523 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.320485 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.320818 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.320559 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.320818 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.320625 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.320818 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.320768 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.320997 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.320969 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.321183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.321046 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.321267 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.321198 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.321267 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.321206 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.321382 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.321299 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4faf119c-25e3-4f4e-80d7-998f0f46a492-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.322740 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.322719 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.323054 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.323033 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.392904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.392836 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp4pb\" (UniqueName: \"kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-kube-api-access-vp4pb\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.395727 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.395694 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-pgvz7\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.463834 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.463803 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:13.952896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.952856 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7"] Nov 04 01:25:13.957633 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:25:13.957607 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4faf119c_25e3_4f4e_80d7_998f0f46a492.slice/crio-b27c4f97772a737ccd4703301745b363ba2c02b33a1b2384ff5096e53d5a6fd0 WatchSource:0}: Error finding container b27c4f97772a737ccd4703301745b363ba2c02b33a1b2384ff5096e53d5a6fd0: Status 404 returned error can't find the container with id b27c4f97772a737ccd4703301745b363ba2c02b33a1b2384ff5096e53d5a6fd0 Nov 04 01:25:13.960234 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.960198 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:13.960317 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.960270 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:13.960356 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:13.960317 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:14.830894 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:14.830860 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" event={"ID":"4faf119c-25e3-4f4e-80d7-998f0f46a492","Type":"ContainerStarted","Data":"d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f"} Nov 04 01:25:14.831071 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:14.830913 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" event={"ID":"4faf119c-25e3-4f4e-80d7-998f0f46a492","Type":"ContainerStarted","Data":"b27c4f97772a737ccd4703301745b363ba2c02b33a1b2384ff5096e53d5a6fd0"} Nov 04 01:25:15.112182 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:25:15.112142 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:25:15.478194 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:15.475389 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:15.478194 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:15.476791 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pgvz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:25:15.478194 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:15.476842 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" podUID="4faf119c-25e3-4f4e-80d7-998f0f46a492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:25:16.470183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:16.468490 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pgvz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:25:16.470183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:16.468563 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" podUID="4faf119c-25e3-4f4e-80d7-998f0f46a492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:25:17.218488 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.217798 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" podStartSLOduration=4.217776812 podStartE2EDuration="4.217776812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:15.051922693 +0000 UTC m=+1660.668268460" watchObservedRunningTime="2026-11-04 01:25:17.217776812 +0000 UTC m=+1662.834122575" Nov 04 01:25:17.237158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.236697 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn"] Nov 04 01:25:17.237903 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.237776 5786 cpu_manager.go:414] "containerMap: removing container" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" Nov 04 01:25:17.237903 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.237802 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3031f39-b918-4fa7-bfc4-b23a14bb83d1" containerName="istio-proxy" Nov 04 01:25:17.340381 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.340343 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn"] Nov 04 01:25:17.340755 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.340731 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" Nov 04 01:25:17.497205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.491275 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pgvz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:25:17.497205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.491331 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" podUID="4faf119c-25e3-4f4e-80d7-998f0f46a492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:25:17.497205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.492329 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f10575b3\"" Nov 04 01:25:17.497205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.492975 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f10575b3-dockercfg-4lxlk\"" Nov 04 01:25:17.569144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.568861 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca567d6-2994-4217-aab6-0ab526410934-plugins-config-volume\") pod \"payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn\" (UID: \"8ca567d6-2994-4217-aab6-0ab526410934\") " pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" Nov 04 01:25:17.569144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.569073 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phwrc\" (UniqueName: \"kubernetes.io/projected/8ca567d6-2994-4217-aab6-0ab526410934-kube-api-access-phwrc\") pod \"payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn\" (UID: \"8ca567d6-2994-4217-aab6-0ab526410934\") " pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" Nov 04 01:25:17.695503 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.675199 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca567d6-2994-4217-aab6-0ab526410934-plugins-config-volume\") pod \"payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn\" (UID: \"8ca567d6-2994-4217-aab6-0ab526410934\") " pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" Nov 04 01:25:17.711904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.697275 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phwrc\" (UniqueName: \"kubernetes.io/projected/8ca567d6-2994-4217-aab6-0ab526410934-kube-api-access-phwrc\") pod \"payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn\" (UID: \"8ca567d6-2994-4217-aab6-0ab526410934\") " pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" Nov 04 01:25:17.712895 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.703768 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca567d6-2994-4217-aab6-0ab526410934-plugins-config-volume\") pod \"payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn\" (UID: \"8ca567d6-2994-4217-aab6-0ab526410934\") " pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" Nov 04 01:25:17.713095 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.706789 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.762030 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.761953 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phwrc\" (UniqueName: \"kubernetes.io/projected/8ca567d6-2994-4217-aab6-0ab526410934-kube-api-access-phwrc\") pod \"payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn\" (UID: \"8ca567d6-2994-4217-aab6-0ab526410934\") " pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" Nov 04 01:25:17.839339 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:17.839313 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" Nov 04 01:25:18.243698 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:18.243661 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn"] Nov 04 01:25:18.248319 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:25:18.248292 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca567d6_2994_4217_aab6_0ab526410934.slice/crio-3163bcbcf540354f64c34af08c1d6e1a1baa4605f9255139e2e01b641587e394 WatchSource:0}: Error finding container 3163bcbcf540354f64c34af08c1d6e1a1baa4605f9255139e2e01b641587e394: Status 404 returned error can't find the container with id 3163bcbcf540354f64c34af08c1d6e1a1baa4605f9255139e2e01b641587e394 Nov 04 01:25:18.393470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:18.393434 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" event={"ID":"8ca567d6-2994-4217-aab6-0ab526410934","Type":"ContainerStarted","Data":"e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481"} Nov 04 01:25:18.393612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:18.393497 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" event={"ID":"8ca567d6-2994-4217-aab6-0ab526410934","Type":"ContainerStarted","Data":"3163bcbcf540354f64c34af08c1d6e1a1baa4605f9255139e2e01b641587e394"} Nov 04 01:25:18.393612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:18.393524 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" Nov 04 01:25:18.476150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:18.471459 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pgvz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:25:18.476150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:18.471520 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" podUID="4faf119c-25e3-4f4e-80d7-998f0f46a492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:25:18.507193 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:18.507074 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" podStartSLOduration=1.507057342 podStartE2EDuration="1.507057342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:18.502418002 +0000 UTC m=+1664.118763765" watchObservedRunningTime="2026-11-04 01:25:18.507057342 +0000 UTC m=+1664.123403106" Nov 04 01:25:19.487264 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:19.487221 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pgvz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:25:19.487460 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:19.487298 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" podUID="4faf119c-25e3-4f4e-80d7-998f0f46a492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:25:20.464127 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:20.464090 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pgvz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:25:20.464530 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:20.464181 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" podUID="4faf119c-25e3-4f4e-80d7-998f0f46a492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:25:21.464578 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:21.464540 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pgvz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:25:21.464970 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:21.464613 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" podUID="4faf119c-25e3-4f4e-80d7-998f0f46a492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:25:22.139770 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.139719 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn"] Nov 04 01:25:22.153763 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.153732 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" Nov 04 01:25:22.211094 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.211009 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn"] Nov 04 01:25:22.236515 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.236482 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d3d6ea20-dockercfg-2r89t\"" Nov 04 01:25:22.238072 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.236928 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d3d6ea20\"" Nov 04 01:25:22.338156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.335359 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twt8g\" (UniqueName: \"kubernetes.io/projected/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-kube-api-access-twt8g\") pod \"payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn\" (UID: \"d77462f5-8fc6-460f-9cac-ca3f3ba0526b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" Nov 04 01:25:22.338156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.335633 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn\" (UID: \"d77462f5-8fc6-460f-9cac-ca3f3ba0526b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" Nov 04 01:25:22.428239 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.424415 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm"] Nov 04 01:25:22.614877 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.614838 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twt8g\" (UniqueName: \"kubernetes.io/projected/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-kube-api-access-twt8g\") pod \"payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn\" (UID: \"d77462f5-8fc6-460f-9cac-ca3f3ba0526b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" Nov 04 01:25:22.615655 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.615628 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn\" (UID: \"d77462f5-8fc6-460f-9cac-ca3f3ba0526b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" Nov 04 01:25:22.617021 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.616998 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn\" (UID: \"d77462f5-8fc6-460f-9cac-ca3f3ba0526b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" Nov 04 01:25:22.654948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.654917 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm"] Nov 04 01:25:22.662580 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.662530 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" Nov 04 01:25:22.691607 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.690214 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twt8g\" (UniqueName: \"kubernetes.io/projected/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-kube-api-access-twt8g\") pod \"payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn\" (UID: \"d77462f5-8fc6-460f-9cac-ca3f3ba0526b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" Nov 04 01:25:22.717062 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.717028 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193ccb6c-79b7-47b9-8614-5fc505f86245-plugins-config-volume\") pod \"payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm\" (UID: \"193ccb6c-79b7-47b9-8614-5fc505f86245\") " pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" Nov 04 01:25:22.717265 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.717155 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkwjr\" (UniqueName: \"kubernetes.io/projected/193ccb6c-79b7-47b9-8614-5fc505f86245-kube-api-access-pkwjr\") pod \"payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm\" (UID: \"193ccb6c-79b7-47b9-8614-5fc505f86245\") " pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" Nov 04 01:25:22.818909 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.818856 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193ccb6c-79b7-47b9-8614-5fc505f86245-plugins-config-volume\") pod \"payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm\" (UID: \"193ccb6c-79b7-47b9-8614-5fc505f86245\") " pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" Nov 04 01:25:22.819140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.819038 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwjr\" (UniqueName: \"kubernetes.io/projected/193ccb6c-79b7-47b9-8614-5fc505f86245-kube-api-access-pkwjr\") pod \"payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm\" (UID: \"193ccb6c-79b7-47b9-8614-5fc505f86245\") " pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" Nov 04 01:25:22.819725 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.819702 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193ccb6c-79b7-47b9-8614-5fc505f86245-plugins-config-volume\") pod \"payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm\" (UID: \"193ccb6c-79b7-47b9-8614-5fc505f86245\") " pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" Nov 04 01:25:22.828037 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.828012 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" Nov 04 01:25:22.875085 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:22.875048 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwjr\" (UniqueName: \"kubernetes.io/projected/193ccb6c-79b7-47b9-8614-5fc505f86245-kube-api-access-pkwjr\") pod \"payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm\" (UID: \"193ccb6c-79b7-47b9-8614-5fc505f86245\") " pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" Nov 04 01:25:23.005810 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.005715 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" Nov 04 01:25:23.227373 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.227339 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn"] Nov 04 01:25:23.228434 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:25:23.228403 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77462f5_8fc6_460f_9cac_ca3f3ba0526b.slice/crio-96809d9eb2bfbe1bb710367998d61b44fdb6097e7b78dd7dd953faab4715135e WatchSource:0}: Error finding container 96809d9eb2bfbe1bb710367998d61b44fdb6097e7b78dd7dd953faab4715135e: Status 404 returned error can't find the container with id 96809d9eb2bfbe1bb710367998d61b44fdb6097e7b78dd7dd953faab4715135e Nov 04 01:25:23.396486 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.396436 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm"] Nov 04 01:25:23.400368 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:25:23.400330 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193ccb6c_79b7_47b9_8614_5fc505f86245.slice/crio-cd717956cb22331408e49fc96714cf6436ece7e42af2d0f5209dc58ae07c7a56 WatchSource:0}: Error finding container cd717956cb22331408e49fc96714cf6436ece7e42af2d0f5209dc58ae07c7a56: Status 404 returned error can't find the container with id cd717956cb22331408e49fc96714cf6436ece7e42af2d0f5209dc58ae07c7a56 Nov 04 01:25:23.546977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.546898 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:23.547452 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.547425 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:23.548445 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.548424 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:25:23.687297 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.687257 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" event={"ID":"193ccb6c-79b7-47b9-8614-5fc505f86245","Type":"ContainerStarted","Data":"1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6"} Nov 04 01:25:23.687722 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.687323 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" Nov 04 01:25:23.687722 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.687346 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" event={"ID":"193ccb6c-79b7-47b9-8614-5fc505f86245","Type":"ContainerStarted","Data":"cd717956cb22331408e49fc96714cf6436ece7e42af2d0f5209dc58ae07c7a56"} Nov 04 01:25:23.688933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.688904 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" event={"ID":"d77462f5-8fc6-460f-9cac-ca3f3ba0526b","Type":"ContainerStarted","Data":"ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3"} Nov 04 01:25:23.689026 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.688953 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" event={"ID":"d77462f5-8fc6-460f-9cac-ca3f3ba0526b","Type":"ContainerStarted","Data":"96809d9eb2bfbe1bb710367998d61b44fdb6097e7b78dd7dd953faab4715135e"} Nov 04 01:25:23.689102 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.689085 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" Nov 04 01:25:23.893590 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:23.893536 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" podStartSLOduration=1.893522191 podStartE2EDuration="1.893522191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:23.889110001 +0000 UTC m=+1669.505455763" watchObservedRunningTime="2026-11-04 01:25:23.893522191 +0000 UTC m=+1669.509867954" Nov 04 01:25:24.063627 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:24.063558 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" podStartSLOduration=2.063538078 podStartE2EDuration="2.063538078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:24.063143841 +0000 UTC m=+1669.679489606" watchObservedRunningTime="2026-11-04 01:25:24.063538078 +0000 UTC m=+1669.679883838" Nov 04 01:25:27.567999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:27.567965 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:25:29.405144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:29.404209 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" Nov 04 01:25:34.699791 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:34.699758 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" Nov 04 01:25:34.700977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:34.700956 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" Nov 04 01:25:45.089202 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:25:45.089159 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:25:54.555407 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.555372 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j"] Nov 04 01:25:54.560102 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.560079 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:54.601142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.601094 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade00d43-25af-4c67-bf66-53fd5a3af630-openshift-service-ca\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:54.601467 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.601445 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-metrics-certs\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:54.601672 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.601650 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-maas-api-tls\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:54.601975 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.601913 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd4cq\" (UniqueName: \"kubernetes.io/projected/ade00d43-25af-4c67-bf66-53fd5a3af630-kube-api-access-xd4cq\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:54.609947 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.609883 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f10575b3-serving-cert\"" Nov 04 01:25:54.656441 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.656405 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j"] Nov 04 01:25:54.706568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.706522 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd4cq\" (UniqueName: \"kubernetes.io/projected/ade00d43-25af-4c67-bf66-53fd5a3af630-kube-api-access-xd4cq\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:54.706763 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.706636 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade00d43-25af-4c67-bf66-53fd5a3af630-openshift-service-ca\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:54.706834 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.706765 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-metrics-certs\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:54.706896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.706869 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-maas-api-tls\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:54.707045 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:25:54.707029 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-serving-cert: secret "maas-api-e2e-ait-f10575b3-serving-cert" not found Nov 04 01:25:54.707187 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:25:54.707091 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-maas-api-tls podName:ade00d43-25af-4c67-bf66-53fd5a3af630 nodeName:}" failed. No retries permitted until 2026-11-04 01:25:55.20707154 +0000 UTC m=+1700.823417301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-maas-api-tls") pod "maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" (UID: "ade00d43-25af-4c67-bf66-53fd5a3af630") : secret "maas-api-e2e-ait-f10575b3-serving-cert" not found Nov 04 01:25:54.707662 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.707629 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade00d43-25af-4c67-bf66-53fd5a3af630-openshift-service-ca\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:54.717174 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.710160 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-metrics-certs\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:54.761980 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:54.761945 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd4cq\" (UniqueName: \"kubernetes.io/projected/ade00d43-25af-4c67-bf66-53fd5a3af630-kube-api-access-xd4cq\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:55.218640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:55.218606 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-maas-api-tls\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:55.221088 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:55.221066 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-maas-api-tls\") pod \"maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:55.473988 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:55.473886 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:55.853294 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:55.853177 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j"] Nov 04 01:25:55.856822 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:25:55.856786 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade00d43_25af_4c67_bf66_53fd5a3af630.slice/crio-71a9bb655ab9bbde582ba86c17d815a8414cb55f2dc347e7d3642b0b459f7fb9 WatchSource:0}: Error finding container 71a9bb655ab9bbde582ba86c17d815a8414cb55f2dc347e7d3642b0b459f7fb9: Status 404 returned error can't find the container with id 71a9bb655ab9bbde582ba86c17d815a8414cb55f2dc347e7d3642b0b459f7fb9 Nov 04 01:25:56.168426 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:56.168347 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" event={"ID":"ade00d43-25af-4c67-bf66-53fd5a3af630","Type":"ContainerStarted","Data":"71a9bb655ab9bbde582ba86c17d815a8414cb55f2dc347e7d3642b0b459f7fb9"} Nov 04 01:25:57.176845 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:57.176811 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" event={"ID":"ade00d43-25af-4c67-bf66-53fd5a3af630","Type":"ContainerStarted","Data":"8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503"} Nov 04 01:25:57.177266 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:57.176883 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:25:57.291145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:57.291050 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" podStartSLOduration=2.337943245 podStartE2EDuration="3.291033799s" totalImagesPullingTime="953.090554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:57.288501398 +0000 UTC m=+1702.904847162" watchObservedRunningTime="2026-11-04 01:25:57.291033799 +0000 UTC m=+1702.907379563" Nov 04 01:25:59.509142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.508171 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d"] Nov 04 01:25:59.639677 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.636349 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d"] Nov 04 01:25:59.639677 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.636423 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj"] Nov 04 01:25:59.662142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.654267 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" Nov 04 01:25:59.713570 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.713536 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj"] Nov 04 01:25:59.713931 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.713865 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" Nov 04 01:25:59.759198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.734719 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-9a0432\"" Nov 04 01:25:59.809680 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.793510 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-9a0432-dockercfg-nzjm7\"" Nov 04 01:25:59.809680 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.798165 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3405d103-6146-4e19-9055-106154b2c39c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj\" (UID: \"3405d103-6146-4e19-9055-106154b2c39c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" Nov 04 01:25:59.809680 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.798260 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj4ls\" (UniqueName: \"kubernetes.io/projected/3405d103-6146-4e19-9055-106154b2c39c-kube-api-access-nj4ls\") pod \"payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj\" (UID: \"3405d103-6146-4e19-9055-106154b2c39c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" Nov 04 01:25:59.809680 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.798568 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc6mq\" (UniqueName: \"kubernetes.io/projected/e3e67e26-14bd-477e-b842-192972b302ab-kube-api-access-lc6mq\") pod \"payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d\" (UID: \"e3e67e26-14bd-477e-b842-192972b302ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" Nov 04 01:25:59.809680 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.798704 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e67e26-14bd-477e-b842-192972b302ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d\" (UID: \"e3e67e26-14bd-477e-b842-192972b302ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" Nov 04 01:25:59.900511 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.900468 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc6mq\" (UniqueName: \"kubernetes.io/projected/e3e67e26-14bd-477e-b842-192972b302ab-kube-api-access-lc6mq\") pod \"payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d\" (UID: \"e3e67e26-14bd-477e-b842-192972b302ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" Nov 04 01:25:59.900713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.900590 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e67e26-14bd-477e-b842-192972b302ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d\" (UID: \"e3e67e26-14bd-477e-b842-192972b302ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" Nov 04 01:25:59.900713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.900702 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3405d103-6146-4e19-9055-106154b2c39c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj\" (UID: \"3405d103-6146-4e19-9055-106154b2c39c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" Nov 04 01:25:59.900858 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.900748 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj4ls\" (UniqueName: \"kubernetes.io/projected/3405d103-6146-4e19-9055-106154b2c39c-kube-api-access-nj4ls\") pod \"payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj\" (UID: \"3405d103-6146-4e19-9055-106154b2c39c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" Nov 04 01:25:59.901434 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.901412 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e67e26-14bd-477e-b842-192972b302ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d\" (UID: \"e3e67e26-14bd-477e-b842-192972b302ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" Nov 04 01:25:59.901520 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.901501 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3405d103-6146-4e19-9055-106154b2c39c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj\" (UID: \"3405d103-6146-4e19-9055-106154b2c39c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" Nov 04 01:25:59.956220 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.956190 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj4ls\" (UniqueName: \"kubernetes.io/projected/3405d103-6146-4e19-9055-106154b2c39c-kube-api-access-nj4ls\") pod \"payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj\" (UID: \"3405d103-6146-4e19-9055-106154b2c39c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" Nov 04 01:25:59.956382 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:25:59.956204 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc6mq\" (UniqueName: \"kubernetes.io/projected/e3e67e26-14bd-477e-b842-192972b302ab-kube-api-access-lc6mq\") pod \"payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d\" (UID: \"e3e67e26-14bd-477e-b842-192972b302ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" Nov 04 01:26:00.145665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.145632 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" Nov 04 01:26:00.149614 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.149582 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" Nov 04 01:26:00.187010 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.186927 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz"] Nov 04 01:26:00.193642 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.193612 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.248793 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.248568 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d3d6ea20-serving-cert\"" Nov 04 01:26:00.250273 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.249948 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz"] Nov 04 01:26:00.307505 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.307473 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n8gr\" (UniqueName: \"kubernetes.io/projected/bca5ab91-9b0b-40c0-b7b6-adce26566764-kube-api-access-2n8gr\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.307678 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.307519 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-maas-api-tls\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.307882 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.307861 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-metrics-certs\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.307987 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.307972 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca5ab91-9b0b-40c0-b7b6-adce26566764-openshift-service-ca\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.409967 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.409872 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca5ab91-9b0b-40c0-b7b6-adce26566764-openshift-service-ca\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.410240 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.410215 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n8gr\" (UniqueName: \"kubernetes.io/projected/bca5ab91-9b0b-40c0-b7b6-adce26566764-kube-api-access-2n8gr\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.410347 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.410271 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-maas-api-tls\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.410406 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:26:00.410386 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-serving-cert: secret "maas-api-e2e-ait-d3d6ea20-serving-cert" not found Nov 04 01:26:00.410460 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.410431 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-metrics-certs\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.410520 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:26:00.410465 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-maas-api-tls podName:bca5ab91-9b0b-40c0-b7b6-adce26566764 nodeName:}" failed. No retries permitted until 2026-11-04 01:26:00.910444536 +0000 UTC m=+1706.526790277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-maas-api-tls") pod "maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" (UID: "bca5ab91-9b0b-40c0-b7b6-adce26566764") : secret "maas-api-e2e-ait-d3d6ea20-serving-cert" not found Nov 04 01:26:00.410815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.410795 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca5ab91-9b0b-40c0-b7b6-adce26566764-openshift-service-ca\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.413043 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.413014 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-metrics-certs\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.515158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.498047 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n8gr\" (UniqueName: \"kubernetes.io/projected/bca5ab91-9b0b-40c0-b7b6-adce26566764-kube-api-access-2n8gr\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.573382 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:26:00.573332 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3405d103_6146_4e19_9055_106154b2c39c.slice/crio-631a6dc6b2fd994a833d0d26a080df1920a2ab41e3ce44948b7bb306f7473932 WatchSource:0}: Error finding container 631a6dc6b2fd994a833d0d26a080df1920a2ab41e3ce44948b7bb306f7473932: Status 404 returned error can't find the container with id 631a6dc6b2fd994a833d0d26a080df1920a2ab41e3ce44948b7bb306f7473932 Nov 04 01:26:00.575323 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.575295 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj"] Nov 04 01:26:00.632158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.631176 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d"] Nov 04 01:26:00.928524 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.928480 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-maas-api-tls\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:00.931002 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:00.930978 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-maas-api-tls\") pod \"maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:01.125171 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:01.125114 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:01.294614 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:01.294574 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" event={"ID":"e3e67e26-14bd-477e-b842-192972b302ab","Type":"ContainerStarted","Data":"ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25"} Nov 04 01:26:01.294819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:01.294636 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" event={"ID":"e3e67e26-14bd-477e-b842-192972b302ab","Type":"ContainerStarted","Data":"5bec5bd585554d9793ccbc980e7e9a89fee07e03e1fb21025b10c71b696a5919"} Nov 04 01:26:01.294819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:01.294668 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" Nov 04 01:26:01.296291 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:01.296262 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" event={"ID":"3405d103-6146-4e19-9055-106154b2c39c","Type":"ContainerStarted","Data":"86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900"} Nov 04 01:26:01.296406 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:01.296321 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" Nov 04 01:26:01.296406 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:01.296348 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" event={"ID":"3405d103-6146-4e19-9055-106154b2c39c","Type":"ContainerStarted","Data":"631a6dc6b2fd994a833d0d26a080df1920a2ab41e3ce44948b7bb306f7473932"} Nov 04 01:26:01.405591 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:01.405487 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" podStartSLOduration=2.405472811 podStartE2EDuration="2.405472811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:01.404306845 +0000 UTC m=+1707.020652607" watchObservedRunningTime="2026-11-04 01:26:01.405472811 +0000 UTC m=+1707.021818607" Nov 04 01:26:01.516889 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:01.516822 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" podStartSLOduration=2.516802276 podStartE2EDuration="2.516802276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:01.512908175 +0000 UTC m=+1707.129253937" watchObservedRunningTime="2026-11-04 01:26:01.516802276 +0000 UTC m=+1707.133148040" Nov 04 01:26:01.605056 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:01.599788 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz"] Nov 04 01:26:02.304736 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:02.304697 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" event={"ID":"bca5ab91-9b0b-40c0-b7b6-adce26566764","Type":"ContainerStarted","Data":"d95c2228c1a27ddca37b2323db7d32a30c476bdca95750d440af81f3c9526aa2"} Nov 04 01:26:03.199838 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:03.199812 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:26:03.313372 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:03.313335 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" event={"ID":"bca5ab91-9b0b-40c0-b7b6-adce26566764","Type":"ContainerStarted","Data":"2e50df3cb74d1dfdfa18f37db50d455f14d7d1d06adc72fdefeca621fb5739db"} Nov 04 01:26:03.313582 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:03.313400 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:03.427522 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:03.427455 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" podStartSLOduration=2.313952895 podStartE2EDuration="3.427437501s" totalImagesPullingTime="1.113484606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:03.423962578 +0000 UTC m=+1709.040308342" watchObservedRunningTime="2026-11-04 01:26:03.427437501 +0000 UTC m=+1709.043783264" Nov 04 01:26:09.346757 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:09.346724 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:26:11.452455 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.451197 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl"] Nov 04 01:26:11.456724 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.456699 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" Nov 04 01:26:11.507059 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.506900 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl"] Nov 04 01:26:11.513907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.513868 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70495ea8-fbc0-4417-933f-14370aaacd8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl\" (UID: \"70495ea8-fbc0-4417-933f-14370aaacd8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" Nov 04 01:26:11.514099 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.514050 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkdqm\" (UniqueName: \"kubernetes.io/projected/70495ea8-fbc0-4417-933f-14370aaacd8d-kube-api-access-gkdqm\") pod \"payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl\" (UID: \"70495ea8-fbc0-4417-933f-14370aaacd8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" Nov 04 01:26:11.549873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.549836 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8aa9f3a2\"" Nov 04 01:26:11.550046 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.549905 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8aa9f3a2-dockercfg-jknb5\"" Nov 04 01:26:11.629142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.628309 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkdqm\" (UniqueName: \"kubernetes.io/projected/70495ea8-fbc0-4417-933f-14370aaacd8d-kube-api-access-gkdqm\") pod \"payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl\" (UID: \"70495ea8-fbc0-4417-933f-14370aaacd8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" Nov 04 01:26:11.629142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.628661 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70495ea8-fbc0-4417-933f-14370aaacd8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl\" (UID: \"70495ea8-fbc0-4417-933f-14370aaacd8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" Nov 04 01:26:11.644139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.641483 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70495ea8-fbc0-4417-933f-14370aaacd8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl\" (UID: \"70495ea8-fbc0-4417-933f-14370aaacd8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" Nov 04 01:26:11.712187 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.706162 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq"] Nov 04 01:26:11.793831 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.785918 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" Nov 04 01:26:11.903268 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.808291 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq"] Nov 04 01:26:11.926666 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.906132 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt2cj\" (UniqueName: \"kubernetes.io/projected/9566986c-b6c8-4785-91c7-34f0193b9b0c-kube-api-access-bt2cj\") pod \"payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq\" (UID: \"9566986c-b6c8-4785-91c7-34f0193b9b0c\") " pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" Nov 04 01:26:11.950144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.930543 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9566986c-b6c8-4785-91c7-34f0193b9b0c-plugins-config-volume\") pod \"payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq\" (UID: \"9566986c-b6c8-4785-91c7-34f0193b9b0c\") " pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" Nov 04 01:26:11.975189 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:11.953193 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkdqm\" (UniqueName: \"kubernetes.io/projected/70495ea8-fbc0-4417-933f-14370aaacd8d-kube-api-access-gkdqm\") pod \"payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl\" (UID: \"70495ea8-fbc0-4417-933f-14370aaacd8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" Nov 04 01:26:12.051924 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:12.051891 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt2cj\" (UniqueName: \"kubernetes.io/projected/9566986c-b6c8-4785-91c7-34f0193b9b0c-kube-api-access-bt2cj\") pod \"payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq\" (UID: \"9566986c-b6c8-4785-91c7-34f0193b9b0c\") " pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" Nov 04 01:26:12.052086 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:12.051976 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9566986c-b6c8-4785-91c7-34f0193b9b0c-plugins-config-volume\") pod \"payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq\" (UID: \"9566986c-b6c8-4785-91c7-34f0193b9b0c\") " pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" Nov 04 01:26:12.052791 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:12.052770 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9566986c-b6c8-4785-91c7-34f0193b9b0c-plugins-config-volume\") pod \"payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq\" (UID: \"9566986c-b6c8-4785-91c7-34f0193b9b0c\") " pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" Nov 04 01:26:12.080810 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:12.080784 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" Nov 04 01:26:12.107249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:12.107222 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt2cj\" (UniqueName: \"kubernetes.io/projected/9566986c-b6c8-4785-91c7-34f0193b9b0c-kube-api-access-bt2cj\") pod \"payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq\" (UID: \"9566986c-b6c8-4785-91c7-34f0193b9b0c\") " pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" Nov 04 01:26:12.308255 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:12.308183 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" Nov 04 01:26:12.308255 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:12.308255 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" Nov 04 01:26:12.311979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:12.311954 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" Nov 04 01:26:12.472582 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:26:12.472553 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70495ea8_fbc0_4417_933f_14370aaacd8d.slice/crio-c77eaadf4a17670db136abee2232bccd2054e68ef83f540fa0da9fa18cb5b46c WatchSource:0}: Error finding container c77eaadf4a17670db136abee2232bccd2054e68ef83f540fa0da9fa18cb5b46c: Status 404 returned error can't find the container with id c77eaadf4a17670db136abee2232bccd2054e68ef83f540fa0da9fa18cb5b46c Nov 04 01:26:12.475532 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:12.475501 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl"] Nov 04 01:26:12.695169 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:12.695111 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq"] Nov 04 01:26:12.698388 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:26:12.698356 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9566986c_b6c8_4785_91c7_34f0193b9b0c.slice/crio-9310b9857644edc4933669afce1c0656e92efa6bdcbcb61a7ffe13750aa79619 WatchSource:0}: Error finding container 9310b9857644edc4933669afce1c0656e92efa6bdcbcb61a7ffe13750aa79619: Status 404 returned error can't find the container with id 9310b9857644edc4933669afce1c0656e92efa6bdcbcb61a7ffe13750aa79619 Nov 04 01:26:13.424034 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:13.423995 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" event={"ID":"70495ea8-fbc0-4417-933f-14370aaacd8d","Type":"ContainerStarted","Data":"240226728b79b7a1e8836462deb843c811ad923fdb1f25e07685f98c44bbf8ea"} Nov 04 01:26:13.424227 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:13.424057 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" Nov 04 01:26:13.424227 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:13.424078 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" event={"ID":"70495ea8-fbc0-4417-933f-14370aaacd8d","Type":"ContainerStarted","Data":"c77eaadf4a17670db136abee2232bccd2054e68ef83f540fa0da9fa18cb5b46c"} Nov 04 01:26:13.425799 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:13.425769 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" event={"ID":"9566986c-b6c8-4785-91c7-34f0193b9b0c","Type":"ContainerStarted","Data":"6328365cb914640034601411e576bf3d86d7fe45f7173dc0788e93f59dff7698"} Nov 04 01:26:13.425924 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:13.425823 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" event={"ID":"9566986c-b6c8-4785-91c7-34f0193b9b0c","Type":"ContainerStarted","Data":"9310b9857644edc4933669afce1c0656e92efa6bdcbcb61a7ffe13750aa79619"} Nov 04 01:26:13.425924 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:13.425851 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" Nov 04 01:26:13.540966 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:13.540900 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" podStartSLOduration=2.540881102 podStartE2EDuration="2.540881102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:13.53653344 +0000 UTC m=+1719.152879202" watchObservedRunningTime="2026-11-04 01:26:13.540881102 +0000 UTC m=+1719.157226866" Nov 04 01:26:13.672229 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:13.650826 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" podStartSLOduration=2.650808331 podStartE2EDuration="2.650808331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:13.64810074 +0000 UTC m=+1719.264446503" watchObservedRunningTime="2026-11-04 01:26:13.650808331 +0000 UTC m=+1719.267154094" Nov 04 01:26:15.088336 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:26:15.088251 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:26:19.328828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:19.328798 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v"] Nov 04 01:26:19.441286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:19.432262 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" Nov 04 01:26:19.441286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:19.406030 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v"] Nov 04 01:26:19.488588 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:19.488549 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-5186fe\"" Nov 04 01:26:19.537139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:19.536542 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-5186fe-dockercfg-kpc8m\"" Nov 04 01:26:19.619245 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:19.619203 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v\" (UID: \"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" Nov 04 01:26:19.619436 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:19.619310 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6t9v\" (UniqueName: \"kubernetes.io/projected/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-kube-api-access-m6t9v\") pod \"payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v\" (UID: \"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" Nov 04 01:26:19.741045 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:19.740999 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v\" (UID: \"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" Nov 04 01:26:19.741389 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:19.741366 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6t9v\" (UniqueName: \"kubernetes.io/projected/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-kube-api-access-m6t9v\") pod \"payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v\" (UID: \"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" Nov 04 01:26:19.787143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:19.750112 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v\" (UID: \"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" Nov 04 01:26:19.855995 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:19.855844 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6t9v\" (UniqueName: \"kubernetes.io/projected/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-kube-api-access-m6t9v\") pod \"payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v\" (UID: \"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" Nov 04 01:26:20.096417 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:20.091451 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" Nov 04 01:26:20.718836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:20.718804 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v"] Nov 04 01:26:20.721415 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:26:20.721383 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a241a8_c4a4_4a4e_9dbc_fbe047089b0c.slice/crio-09b2a40bbfc1ce0c79d7e4f08d667fba1ab5247d60e0a2329b63d4a60c3ac0b1 WatchSource:0}: Error finding container 09b2a40bbfc1ce0c79d7e4f08d667fba1ab5247d60e0a2329b63d4a60c3ac0b1: Status 404 returned error can't find the container with id 09b2a40bbfc1ce0c79d7e4f08d667fba1ab5247d60e0a2329b63d4a60c3ac0b1 Nov 04 01:26:21.554877 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:21.554839 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" event={"ID":"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c","Type":"ContainerStarted","Data":"281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74"} Nov 04 01:26:21.555065 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:21.554898 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" event={"ID":"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c","Type":"ContainerStarted","Data":"09b2a40bbfc1ce0c79d7e4f08d667fba1ab5247d60e0a2329b63d4a60c3ac0b1"} Nov 04 01:26:21.555065 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:21.554939 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" Nov 04 01:26:21.711146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:21.703737 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" podStartSLOduration=2.70371717 podStartE2EDuration="2.70371717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:21.699434828 +0000 UTC m=+1727.315780592" watchObservedRunningTime="2026-11-04 01:26:21.70371717 +0000 UTC m=+1727.320062934" Nov 04 01:26:24.438104 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:24.438071 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" Nov 04 01:26:24.439203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:24.439185 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" Nov 04 01:26:27.808333 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:27.808293 5786 patch_prober.go:29] interesting pod/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:26:27.808821 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:27.808374 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:28.813296 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:28.813250 5786 patch_prober.go:29] interesting pod/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:26:28.813828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:28.813337 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:30.702685 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:30.702641 5786 generic.go:356] "Generic (PLEG): container finished" podID="8e1bf62f-188c-4b74-876d-da9212788972" containerID="d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84" exitCode=0 Nov 04 01:26:30.703243 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:30.702714 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" event={"ID":"8e1bf62f-188c-4b74-876d-da9212788972","Type":"ContainerDied","Data":"d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84"} Nov 04 01:26:30.703340 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:30.703322 5786 scope.go:122] "RemoveContainer" containerID="d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84" Nov 04 01:26:30.705071 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:30.705040 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:26:30.705184 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:30.705112 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:26:30.705184 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:30.705177 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:26:31.712014 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:31.711977 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" event={"ID":"8e1bf62f-188c-4b74-876d-da9212788972","Type":"ContainerStarted","Data":"ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1"} Nov 04 01:26:31.713849 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:31.713823 5786 generic.go:356] "Generic (PLEG): container finished" podID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerID="7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf" exitCode=0 Nov 04 01:26:31.713984 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:31.713857 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" event={"ID":"7181de9e-3d8a-409e-bdd8-de4afd7b4be1","Type":"ContainerDied","Data":"7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf"} Nov 04 01:26:31.714311 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:31.714293 5786 scope.go:122] "RemoveContainer" containerID="7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf" Nov 04 01:26:31.715962 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:31.715930 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:26:31.716030 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:31.716001 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:26:31.716080 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:31.716033 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:26:31.813043 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:31.813017 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:26:31.813295 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:31.813273 5786 patch_prober.go:29] interesting pod/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:26:31.813349 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:31.813319 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:26:32.635317 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:32.633734 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" Nov 04 01:26:32.804142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:32.802245 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" event={"ID":"7181de9e-3d8a-409e-bdd8-de4afd7b4be1","Type":"ContainerStarted","Data":"0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275"} Nov 04 01:26:32.822141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:32.820788 5786 patch_prober.go:29] interesting pod/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:26:32.822141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:32.820844 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:26:33.198144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:33.197339 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:26:33.198144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:33.197505 5786 patch_prober.go:29] interesting pod/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:26:33.198144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:33.197562 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:26:33.815141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:33.814687 5786 patch_prober.go:29] interesting pod/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:26:33.815141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:33.814773 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:26:34.204010 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:34.203971 5786 patch_prober.go:29] interesting pod/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:26:34.204811 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:34.204777 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:26:34.814614 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:34.814583 5786 patch_prober.go:29] interesting pod/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:26:34.814896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:34.814870 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:26:35.199617 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:35.199576 5786 patch_prober.go:29] interesting pod/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:26:35.200263 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:35.200220 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:26:35.813823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:35.813378 5786 patch_prober.go:29] interesting pod/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:26:35.813823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:35.813447 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:26:36.200566 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:36.200522 5786 patch_prober.go:29] interesting pod/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:26:36.201944 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:36.201911 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:26:36.813921 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:36.813881 5786 patch_prober.go:29] interesting pod/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:26:36.814251 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:36.814219 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:26:37.198625 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:37.198582 5786 patch_prober.go:29] interesting pod/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:26:37.198937 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:37.198911 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:26:37.818374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:37.818341 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:26:37.818916 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:37.818895 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:26:37.819841 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:37.819808 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:26:38.204620 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:38.204581 5786 patch_prober.go:29] interesting pod/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:26:38.204832 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:38.204658 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:26:39.201942 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:39.201907 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:26:39.202466 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:39.202400 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:26:39.203237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:39.203215 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:26:45.090009 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:26:45.089963 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:26:57.127361 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.127324 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq"] Nov 04 01:26:57.131035 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.131015 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.165872 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.165833 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-metrics-certs\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.165872 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.165874 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-maas-api-tls\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.166147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.166037 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvfpq\" (UniqueName: \"kubernetes.io/projected/b4447b9a-810f-4410-909c-9d51764eff35-kube-api-access-gvfpq\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.166147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.166081 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4447b9a-810f-4410-909c-9d51764eff35-openshift-service-ca\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.182044 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.182014 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8aa9f3a2-serving-cert\"" Nov 04 01:26:57.233719 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.233683 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq"] Nov 04 01:26:57.267774 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.267733 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvfpq\" (UniqueName: \"kubernetes.io/projected/b4447b9a-810f-4410-909c-9d51764eff35-kube-api-access-gvfpq\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.267965 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.267795 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4447b9a-810f-4410-909c-9d51764eff35-openshift-service-ca\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.268467 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.268440 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-metrics-certs\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.269094 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.268479 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-maas-api-tls\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.269094 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:26:57.268642 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-serving-cert: secret "maas-api-e2e-mt-8aa9f3a2-serving-cert" not found Nov 04 01:26:57.269094 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:26:57.268711 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-maas-api-tls podName:b4447b9a-810f-4410-909c-9d51764eff35 nodeName:}" failed. No retries permitted until 2026-11-04 01:26:57.768691611 +0000 UTC m=+1763.385037371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-maas-api-tls") pod "maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" (UID: "b4447b9a-810f-4410-909c-9d51764eff35") : secret "maas-api-e2e-mt-8aa9f3a2-serving-cert" not found Nov 04 01:26:57.269094 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.268863 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4447b9a-810f-4410-909c-9d51764eff35-openshift-service-ca\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.271554 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.271532 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-metrics-certs\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.324383 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.324345 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvfpq\" (UniqueName: \"kubernetes.io/projected/b4447b9a-810f-4410-909c-9d51764eff35-kube-api-access-gvfpq\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.781989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.781954 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-maas-api-tls\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:57.784547 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:57.784522 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-maas-api-tls\") pod \"maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:58.044907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:58.044804 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:26:58.425465 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:58.425432 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq"] Nov 04 01:26:58.429856 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:26:58.429825 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4447b9a_810f_4410_909c_9d51764eff35.slice/crio-ab9f842db2ba07bf0a42f6c8b2fb97b9c6427b9e9b9ac824ff08a7b140411f05 WatchSource:0}: Error finding container ab9f842db2ba07bf0a42f6c8b2fb97b9c6427b9e9b9ac824ff08a7b140411f05: Status 404 returned error can't find the container with id ab9f842db2ba07bf0a42f6c8b2fb97b9c6427b9e9b9ac824ff08a7b140411f05 Nov 04 01:26:59.076965 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:26:59.076933 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" event={"ID":"b4447b9a-810f-4410-909c-9d51764eff35","Type":"ContainerStarted","Data":"ab9f842db2ba07bf0a42f6c8b2fb97b9c6427b9e9b9ac824ff08a7b140411f05"} Nov 04 01:27:00.095962 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:00.095922 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" event={"ID":"b4447b9a-810f-4410-909c-9d51764eff35","Type":"ContainerStarted","Data":"bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5"} Nov 04 01:27:00.096416 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:00.096161 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:27:00.218753 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:00.218688 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" podStartSLOduration=2.286597238 podStartE2EDuration="3.218670074s" totalImagesPullingTime="932.072836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:00.218385673 +0000 UTC m=+1765.834731435" watchObservedRunningTime="2026-11-04 01:27:00.218670074 +0000 UTC m=+1765.835015835" Nov 04 01:27:06.110346 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:06.110300 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:27:07.672970 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:07.672931 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z"] Nov 04 01:27:08.464102 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:08.464054 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:27:13.490080 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:13.490041 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn"] Nov 04 01:27:15.457183 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:27:15.455348 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:27:18.490714 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:18.490669 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" podUID="8ca567d6-2994-4217-aab6-0ab526410934" containerName="payload-processing" containerID="cri-o://e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481" gracePeriod=25 Nov 04 01:27:18.770299 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:18.770273 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" Nov 04 01:27:18.905996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:18.905963 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca567d6-2994-4217-aab6-0ab526410934-kube-api-access-phwrc\" (UniqueName: \"kubernetes.io/projected/8ca567d6-2994-4217-aab6-0ab526410934-kube-api-access-phwrc\") pod \"8ca567d6-2994-4217-aab6-0ab526410934\" (UID: \"8ca567d6-2994-4217-aab6-0ab526410934\") " Nov 04 01:27:18.906203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:18.906077 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca567d6-2994-4217-aab6-0ab526410934-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca567d6-2994-4217-aab6-0ab526410934-plugins-config-volume\") pod \"8ca567d6-2994-4217-aab6-0ab526410934\" (UID: \"8ca567d6-2994-4217-aab6-0ab526410934\") " Nov 04 01:27:18.906494 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:18.906467 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca567d6-2994-4217-aab6-0ab526410934-plugins-config-volume" pod "8ca567d6-2994-4217-aab6-0ab526410934" (UID: "8ca567d6-2994-4217-aab6-0ab526410934"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:18.906803 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:18.906783 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca567d6-2994-4217-aab6-0ab526410934-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:18.908696 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:18.908670 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca567d6-2994-4217-aab6-0ab526410934-kube-api-access-phwrc" pod "8ca567d6-2994-4217-aab6-0ab526410934" (UID: "8ca567d6-2994-4217-aab6-0ab526410934"). InnerVolumeSpecName "kube-api-access-phwrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:19.008539 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:19.008453 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phwrc\" (UniqueName: \"kubernetes.io/projected/8ca567d6-2994-4217-aab6-0ab526410934-kube-api-access-phwrc\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:19.393624 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:19.393582 5786 generic.go:356] "Generic (PLEG): container finished" podID="8ca567d6-2994-4217-aab6-0ab526410934" containerID="e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481" exitCode=0 Nov 04 01:27:19.393846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:19.393648 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" event={"ID":"8ca567d6-2994-4217-aab6-0ab526410934","Type":"ContainerDied","Data":"e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481"} Nov 04 01:27:19.393846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:19.393715 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" event={"ID":"8ca567d6-2994-4217-aab6-0ab526410934","Type":"ContainerDied","Data":"3163bcbcf540354f64c34af08c1d6e1a1baa4605f9255139e2e01b641587e394"} Nov 04 01:27:19.393846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:19.393650 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn" Nov 04 01:27:19.393846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:19.393744 5786 scope.go:122] "RemoveContainer" containerID="e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481" Nov 04 01:27:19.404694 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:19.404673 5786 scope.go:122] "RemoveContainer" containerID="e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481" Nov 04 01:27:19.405020 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:27:19.404996 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481\": container with ID starting with e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481 not found: ID does not exist" containerID="e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481" Nov 04 01:27:19.405066 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:19.405031 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481"} err="failed to get container status \"e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481\": rpc error: code = NotFound desc = could not find container \"e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481\": container with ID starting with e60bac88c5908a1a2186e2e341db61035fe29cd1ea31b9397cba1510adff8481 not found: ID does not exist" Nov 04 01:27:19.559495 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:19.559462 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn"] Nov 04 01:27:19.564714 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:19.564688 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f10575b3-5c8f5b5b6d-flqdn"] Nov 04 01:27:20.507506 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:20.507473 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca567d6-2994-4217-aab6-0ab526410934" path="/var/lib/kubelet/pods/8ca567d6-2994-4217-aab6-0ab526410934/volumes" Nov 04 01:27:22.696299 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:22.696231 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" containerID="cri-o://3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307" gracePeriod=45 Nov 04 01:27:22.783163 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:22.783106 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": dial tcp 10.132.0.52:8000: connect: connection refused" Nov 04 01:27:22.978954 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:22.978923 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:27:23.086066 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.086030 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-home\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-home\") pod \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " Nov 04 01:27:23.086241 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.086181 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tls-certs\") pod \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " Nov 04 01:27:23.086326 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.086247 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tmp-dir\") pod \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " Nov 04 01:27:23.086374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.086352 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-model-cache\") pod \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " Nov 04 01:27:23.086411 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.086370 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-home" pod "f457f48b-d5aa-43d0-ab4d-0df141a949c1" (UID: "f457f48b-d5aa-43d0-ab4d-0df141a949c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:23.086506 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.086490 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-dshm\") pod \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " Nov 04 01:27:23.086573 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.086514 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tmp-dir" pod "f457f48b-d5aa-43d0-ab4d-0df141a949c1" (UID: "f457f48b-d5aa-43d0-ab4d-0df141a949c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:23.086573 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.086546 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f457f48b-d5aa-43d0-ab4d-0df141a949c1-kube-api-access-7b72p\" (UniqueName: \"kubernetes.io/projected/f457f48b-d5aa-43d0-ab4d-0df141a949c1-kube-api-access-7b72p\") pod \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\" (UID: \"f457f48b-d5aa-43d0-ab4d-0df141a949c1\") " Nov 04 01:27:23.086736 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.086712 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-model-cache" pod "f457f48b-d5aa-43d0-ab4d-0df141a949c1" (UID: "f457f48b-d5aa-43d0-ab4d-0df141a949c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:23.087418 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.087387 5786 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tmp-dir\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:23.087418 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.087418 5786 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-model-cache\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:23.087600 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.087435 5786 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-home\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:23.088715 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.088691 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tls-certs" pod "f457f48b-d5aa-43d0-ab4d-0df141a949c1" (UID: "f457f48b-d5aa-43d0-ab4d-0df141a949c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:23.088820 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.088740 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-dshm" pod "f457f48b-d5aa-43d0-ab4d-0df141a949c1" (UID: "f457f48b-d5aa-43d0-ab4d-0df141a949c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:23.088935 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.088915 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f457f48b-d5aa-43d0-ab4d-0df141a949c1-kube-api-access-7b72p" pod "f457f48b-d5aa-43d0-ab4d-0df141a949c1" (UID: "f457f48b-d5aa-43d0-ab4d-0df141a949c1"). InnerVolumeSpecName "kube-api-access-7b72p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:23.189166 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.189088 5786 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f457f48b-d5aa-43d0-ab4d-0df141a949c1-dshm\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:23.189166 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.189148 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b72p\" (UniqueName: \"kubernetes.io/projected/f457f48b-d5aa-43d0-ab4d-0df141a949c1-kube-api-access-7b72p\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:23.189166 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.189163 5786 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f457f48b-d5aa-43d0-ab4d-0df141a949c1-tls-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:23.424267 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.424231 5786 generic.go:356] "Generic (PLEG): container finished" podID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerID="3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307" exitCode=0 Nov 04 01:27:23.424468 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.424277 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" Nov 04 01:27:23.424468 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.424311 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" event={"ID":"f457f48b-d5aa-43d0-ab4d-0df141a949c1","Type":"ContainerDied","Data":"3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307"} Nov 04 01:27:23.424468 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.424357 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z" event={"ID":"f457f48b-d5aa-43d0-ab4d-0df141a949c1","Type":"ContainerDied","Data":"0eb9bc1e946e76feb9838690fed52e3a75e17abb3507f6802fd0216bcdc4aac1"} Nov 04 01:27:23.424468 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.424390 5786 scope.go:122] "RemoveContainer" containerID="3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307" Nov 04 01:27:23.438015 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.437991 5786 scope.go:122] "RemoveContainer" containerID="3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307" Nov 04 01:27:23.438388 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:27:23.438369 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307\": container with ID starting with 3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307 not found: ID does not exist" containerID="3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307" Nov 04 01:27:23.438442 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.438396 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307"} err="failed to get container status \"3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307\": rpc error: code = NotFound desc = could not find container \"3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307\": container with ID starting with 3729571fe28659a0bfd2cc8552a30d97993000387f7fd906e92692fc8015a307 not found: ID does not exist" Nov 04 01:27:23.589863 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.589826 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z"] Nov 04 01:27:23.633186 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:23.633148 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2ee045/simulator-w3-2ee045-kserve-bf9999b79-hjb8z"] Nov 04 01:27:24.508020 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:24.507970 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" path="/var/lib/kubelet/pods/f457f48b-d5aa-43d0-ab4d-0df141a949c1/volumes" Nov 04 01:27:30.345801 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:30.345763 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm"] Nov 04 01:27:30.451189 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:30.451154 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn"] Nov 04 01:27:34.736415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:34.736372 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:27:34.798657 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:34.798631 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:27:35.346162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.346086 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" podUID="193ccb6c-79b7-47b9-8614-5fc505f86245" containerName="payload-processing" containerID="cri-o://1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6" gracePeriod=25 Nov 04 01:27:35.452320 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.452234 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" podUID="d77462f5-8fc6-460f-9cac-ca3f3ba0526b" containerName="payload-pre-processing" containerID="cri-o://ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3" gracePeriod=25 Nov 04 01:27:35.639680 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.639267 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" Nov 04 01:27:35.682078 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.682044 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193ccb6c-79b7-47b9-8614-5fc505f86245-kube-api-access-pkwjr\" (UniqueName: \"kubernetes.io/projected/193ccb6c-79b7-47b9-8614-5fc505f86245-kube-api-access-pkwjr\") pod \"193ccb6c-79b7-47b9-8614-5fc505f86245\" (UID: \"193ccb6c-79b7-47b9-8614-5fc505f86245\") " Nov 04 01:27:35.682293 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.682257 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193ccb6c-79b7-47b9-8614-5fc505f86245-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193ccb6c-79b7-47b9-8614-5fc505f86245-plugins-config-volume\") pod \"193ccb6c-79b7-47b9-8614-5fc505f86245\" (UID: \"193ccb6c-79b7-47b9-8614-5fc505f86245\") " Nov 04 01:27:35.682821 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.682791 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193ccb6c-79b7-47b9-8614-5fc505f86245-plugins-config-volume" pod "193ccb6c-79b7-47b9-8614-5fc505f86245" (UID: "193ccb6c-79b7-47b9-8614-5fc505f86245"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:35.683214 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.683189 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193ccb6c-79b7-47b9-8614-5fc505f86245-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:35.685095 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.685064 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193ccb6c-79b7-47b9-8614-5fc505f86245-kube-api-access-pkwjr" pod "193ccb6c-79b7-47b9-8614-5fc505f86245" (UID: "193ccb6c-79b7-47b9-8614-5fc505f86245"). InnerVolumeSpecName "kube-api-access-pkwjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:35.725427 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.725397 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" Nov 04 01:27:35.750545 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.750505 5786 generic.go:356] "Generic (PLEG): container finished" podID="d77462f5-8fc6-460f-9cac-ca3f3ba0526b" containerID="ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3" exitCode=0 Nov 04 01:27:35.750996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.750555 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" Nov 04 01:27:35.750996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.750587 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" event={"ID":"d77462f5-8fc6-460f-9cac-ca3f3ba0526b","Type":"ContainerDied","Data":"ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3"} Nov 04 01:27:35.750996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.750684 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn" event={"ID":"d77462f5-8fc6-460f-9cac-ca3f3ba0526b","Type":"ContainerDied","Data":"96809d9eb2bfbe1bb710367998d61b44fdb6097e7b78dd7dd953faab4715135e"} Nov 04 01:27:35.750996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.750717 5786 scope.go:122] "RemoveContainer" containerID="ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3" Nov 04 01:27:35.758165 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.753478 5786 generic.go:356] "Generic (PLEG): container finished" podID="193ccb6c-79b7-47b9-8614-5fc505f86245" containerID="1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6" exitCode=0 Nov 04 01:27:35.758165 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.753522 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" event={"ID":"193ccb6c-79b7-47b9-8614-5fc505f86245","Type":"ContainerDied","Data":"1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6"} Nov 04 01:27:35.758165 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.753534 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" Nov 04 01:27:35.758165 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.753562 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm" event={"ID":"193ccb6c-79b7-47b9-8614-5fc505f86245","Type":"ContainerDied","Data":"cd717956cb22331408e49fc96714cf6436ece7e42af2d0f5209dc58ae07c7a56"} Nov 04 01:27:35.766266 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.766244 5786 scope.go:122] "RemoveContainer" containerID="ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3" Nov 04 01:27:35.766640 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:27:35.766611 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3\": container with ID starting with ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3 not found: ID does not exist" containerID="ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3" Nov 04 01:27:35.766739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.766651 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3"} err="failed to get container status \"ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3\": rpc error: code = NotFound desc = could not find container \"ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3\": container with ID starting with ebe5d9cdde998ae4d75d9f7f21136378e2516472ad5edc87e34caea77b61f4f3 not found: ID does not exist" Nov 04 01:27:35.766739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.766673 5786 scope.go:122] "RemoveContainer" containerID="1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6" Nov 04 01:27:35.778740 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.778714 5786 scope.go:122] "RemoveContainer" containerID="1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6" Nov 04 01:27:35.779104 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:27:35.779075 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6\": container with ID starting with 1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6 not found: ID does not exist" containerID="1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6" Nov 04 01:27:35.779201 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.779156 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6"} err="failed to get container status \"1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6\": rpc error: code = NotFound desc = could not find container \"1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6\": container with ID starting with 1efe316c6575f76290994a1acc2ddfe0dafca449c31e75f40539f6139a60eaf6 not found: ID does not exist" Nov 04 01:27:35.784847 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.784817 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-kube-api-access-twt8g\" (UniqueName: \"kubernetes.io/projected/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-kube-api-access-twt8g\") pod \"d77462f5-8fc6-460f-9cac-ca3f3ba0526b\" (UID: \"d77462f5-8fc6-460f-9cac-ca3f3ba0526b\") " Nov 04 01:27:35.784993 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.784975 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-plugins-config-volume\") pod \"d77462f5-8fc6-460f-9cac-ca3f3ba0526b\" (UID: \"d77462f5-8fc6-460f-9cac-ca3f3ba0526b\") " Nov 04 01:27:35.785368 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.785342 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-plugins-config-volume" pod "d77462f5-8fc6-460f-9cac-ca3f3ba0526b" (UID: "d77462f5-8fc6-460f-9cac-ca3f3ba0526b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:35.786005 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.785905 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:35.786005 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.785932 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkwjr\" (UniqueName: \"kubernetes.io/projected/193ccb6c-79b7-47b9-8614-5fc505f86245-kube-api-access-pkwjr\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:35.787576 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.787551 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-kube-api-access-twt8g" pod "d77462f5-8fc6-460f-9cac-ca3f3ba0526b" (UID: "d77462f5-8fc6-460f-9cac-ca3f3ba0526b"). InnerVolumeSpecName "kube-api-access-twt8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:35.887921 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.887839 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twt8g\" (UniqueName: \"kubernetes.io/projected/d77462f5-8fc6-460f-9cac-ca3f3ba0526b-kube-api-access-twt8g\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:35.977566 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.977532 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm"] Nov 04 01:27:35.981162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:35.981137 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d3d6ea20-56f86657d4-r4blm"] Nov 04 01:27:36.169143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:36.169040 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn"] Nov 04 01:27:36.174877 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:36.174851 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d3d6ea20-77747f4c6c-l57mn"] Nov 04 01:27:36.508221 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:36.508139 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193ccb6c-79b7-47b9-8614-5fc505f86245" path="/var/lib/kubelet/pods/193ccb6c-79b7-47b9-8614-5fc505f86245/volumes" Nov 04 01:27:36.508707 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:36.508691 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77462f5-8fc6-460f-9cac-ca3f3ba0526b" path="/var/lib/kubelet/pods/d77462f5-8fc6-460f-9cac-ca3f3ba0526b/volumes" Nov 04 01:27:45.089832 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:27:45.089748 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:27:55.949943 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:55.949900 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j"] Nov 04 01:27:55.950732 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:55.950699 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" podUID="ade00d43-25af-4c67-bf66-53fd5a3af630" containerName="maas-api" containerID="cri-o://8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503" gracePeriod=30 Nov 04 01:27:56.939885 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:56.939855 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:27:57.007142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:56.980405 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-metrics-certs\") pod \"ade00d43-25af-4c67-bf66-53fd5a3af630\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " Nov 04 01:27:57.007142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:56.980540 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade00d43-25af-4c67-bf66-53fd5a3af630-kube-api-access-xd4cq\" (UniqueName: \"kubernetes.io/projected/ade00d43-25af-4c67-bf66-53fd5a3af630-kube-api-access-xd4cq\") pod \"ade00d43-25af-4c67-bf66-53fd5a3af630\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " Nov 04 01:27:57.007142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:56.980658 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-maas-api-tls\") pod \"ade00d43-25af-4c67-bf66-53fd5a3af630\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " Nov 04 01:27:57.007142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.001511 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-metrics-certs" pod "ade00d43-25af-4c67-bf66-53fd5a3af630" (UID: "ade00d43-25af-4c67-bf66-53fd5a3af630"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:57.007142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.004331 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade00d43-25af-4c67-bf66-53fd5a3af630-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade00d43-25af-4c67-bf66-53fd5a3af630-openshift-service-ca\") pod \"ade00d43-25af-4c67-bf66-53fd5a3af630\" (UID: \"ade00d43-25af-4c67-bf66-53fd5a3af630\") " Nov 04 01:27:57.007142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.005224 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:57.024140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.022399 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade00d43-25af-4c67-bf66-53fd5a3af630-openshift-service-ca" pod "ade00d43-25af-4c67-bf66-53fd5a3af630" (UID: "ade00d43-25af-4c67-bf66-53fd5a3af630"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:57.038177 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.031464 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-maas-api-tls" pod "ade00d43-25af-4c67-bf66-53fd5a3af630" (UID: "ade00d43-25af-4c67-bf66-53fd5a3af630"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:57.043213 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.043168 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade00d43-25af-4c67-bf66-53fd5a3af630-kube-api-access-xd4cq" pod "ade00d43-25af-4c67-bf66-53fd5a3af630" (UID: "ade00d43-25af-4c67-bf66-53fd5a3af630"). InnerVolumeSpecName "kube-api-access-xd4cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:57.124138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.107607 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ade00d43-25af-4c67-bf66-53fd5a3af630-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:57.139696 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.139667 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade00d43-25af-4c67-bf66-53fd5a3af630-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:57.139978 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.139955 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd4cq\" (UniqueName: \"kubernetes.io/projected/ade00d43-25af-4c67-bf66-53fd5a3af630-kube-api-access-xd4cq\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:57.339158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.336557 5786 generic.go:356] "Generic (PLEG): container finished" podID="ade00d43-25af-4c67-bf66-53fd5a3af630" containerID="8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503" exitCode=0 Nov 04 01:27:57.339158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.336725 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" event={"ID":"ade00d43-25af-4c67-bf66-53fd5a3af630","Type":"ContainerDied","Data":"8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503"} Nov 04 01:27:57.339158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.336781 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" event={"ID":"ade00d43-25af-4c67-bf66-53fd5a3af630","Type":"ContainerDied","Data":"71a9bb655ab9bbde582ba86c17d815a8414cb55f2dc347e7d3642b0b459f7fb9"} Nov 04 01:27:57.339158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.336813 5786 scope.go:122] "RemoveContainer" containerID="8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503" Nov 04 01:27:57.339158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.336986 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j" Nov 04 01:27:57.390114 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.390087 5786 scope.go:122] "RemoveContainer" containerID="8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503" Nov 04 01:27:57.390673 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:27:57.390641 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503\": container with ID starting with 8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503 not found: ID does not exist" containerID="8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503" Nov 04 01:27:57.390741 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.390689 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503"} err="failed to get container status \"8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503\": rpc error: code = NotFound desc = could not find container \"8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503\": container with ID starting with 8339ca8d4a76352672479fd02d6e99b0d2f65ae8261da2b078d897107d887503 not found: ID does not exist" Nov 04 01:27:57.521347 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.521312 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j"] Nov 04 01:27:57.569330 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:57.569282 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f10575b3-774d67c78d-9wq5j"] Nov 04 01:27:58.508188 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:27:58.508143 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade00d43-25af-4c67-bf66-53fd5a3af630" path="/var/lib/kubelet/pods/ade00d43-25af-4c67-bf66-53fd5a3af630/volumes" Nov 04 01:28:00.096933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:00.096893 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn"] Nov 04 01:28:00.097444 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:00.097164 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" containerID="cri-o://ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1" gracePeriod=30 Nov 04 01:28:03.162202 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:03.161050 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz"] Nov 04 01:28:03.162202 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:03.161319 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" podUID="bca5ab91-9b0b-40c0-b7b6-adce26566764" containerName="maas-api" containerID="cri-o://2e50df3cb74d1dfdfa18f37db50d455f14d7d1d06adc72fdefeca621fb5739db" gracePeriod=30 Nov 04 01:28:03.590479 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:03.590415 5786 generic.go:356] "Generic (PLEG): container finished" podID="bca5ab91-9b0b-40c0-b7b6-adce26566764" containerID="2e50df3cb74d1dfdfa18f37db50d455f14d7d1d06adc72fdefeca621fb5739db" exitCode=0 Nov 04 01:28:03.591014 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:03.590969 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" event={"ID":"bca5ab91-9b0b-40c0-b7b6-adce26566764","Type":"ContainerDied","Data":"2e50df3cb74d1dfdfa18f37db50d455f14d7d1d06adc72fdefeca621fb5739db"} Nov 04 01:28:04.076096 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.076067 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:28:04.099429 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.099394 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca5ab91-9b0b-40c0-b7b6-adce26566764-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca5ab91-9b0b-40c0-b7b6-adce26566764-openshift-service-ca\") pod \"bca5ab91-9b0b-40c0-b7b6-adce26566764\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " Nov 04 01:28:04.099625 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.099505 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-metrics-certs\") pod \"bca5ab91-9b0b-40c0-b7b6-adce26566764\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " Nov 04 01:28:04.099625 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.099608 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca5ab91-9b0b-40c0-b7b6-adce26566764-kube-api-access-2n8gr\" (UniqueName: \"kubernetes.io/projected/bca5ab91-9b0b-40c0-b7b6-adce26566764-kube-api-access-2n8gr\") pod \"bca5ab91-9b0b-40c0-b7b6-adce26566764\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " Nov 04 01:28:04.099753 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.099705 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-maas-api-tls\") pod \"bca5ab91-9b0b-40c0-b7b6-adce26566764\" (UID: \"bca5ab91-9b0b-40c0-b7b6-adce26566764\") " Nov 04 01:28:04.099865 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.099827 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca5ab91-9b0b-40c0-b7b6-adce26566764-openshift-service-ca" pod "bca5ab91-9b0b-40c0-b7b6-adce26566764" (UID: "bca5ab91-9b0b-40c0-b7b6-adce26566764"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:04.100418 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.100399 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca5ab91-9b0b-40c0-b7b6-adce26566764-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:04.101709 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.101685 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-metrics-certs" pod "bca5ab91-9b0b-40c0-b7b6-adce26566764" (UID: "bca5ab91-9b0b-40c0-b7b6-adce26566764"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:04.101877 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.101844 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-maas-api-tls" pod "bca5ab91-9b0b-40c0-b7b6-adce26566764" (UID: "bca5ab91-9b0b-40c0-b7b6-adce26566764"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:04.101967 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.101848 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca5ab91-9b0b-40c0-b7b6-adce26566764-kube-api-access-2n8gr" pod "bca5ab91-9b0b-40c0-b7b6-adce26566764" (UID: "bca5ab91-9b0b-40c0-b7b6-adce26566764"). InnerVolumeSpecName "kube-api-access-2n8gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:04.257146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.255691 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:04.257146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.255723 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca5ab91-9b0b-40c0-b7b6-adce26566764-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:04.257146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.255739 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n8gr\" (UniqueName: \"kubernetes.io/projected/bca5ab91-9b0b-40c0-b7b6-adce26566764-kube-api-access-2n8gr\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:04.599155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.599093 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" Nov 04 01:28:04.599334 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.599161 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz" event={"ID":"bca5ab91-9b0b-40c0-b7b6-adce26566764","Type":"ContainerDied","Data":"d95c2228c1a27ddca37b2323db7d32a30c476bdca95750d440af81f3c9526aa2"} Nov 04 01:28:04.599334 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.599233 5786 scope.go:122] "RemoveContainer" containerID="2e50df3cb74d1dfdfa18f37db50d455f14d7d1d06adc72fdefeca621fb5739db" Nov 04 01:28:04.768197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.768163 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz"] Nov 04 01:28:04.815520 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:04.815444 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d3d6ea20-6855545686-cv4tz"] Nov 04 01:28:05.387158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.387133 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:28:05.478416 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.478385 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e1bf62f-188c-4b74-876d-da9212788972-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1bf62f-188c-4b74-876d-da9212788972-istio-podinfo\") pod \"8e1bf62f-188c-4b74-876d-da9212788972\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " Nov 04 01:28:05.478595 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.478567 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-data\") pod \"8e1bf62f-188c-4b74-876d-da9212788972\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " Nov 04 01:28:05.478703 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.478690 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-credential-socket\") pod \"8e1bf62f-188c-4b74-876d-da9212788972\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " Nov 04 01:28:05.478745 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.478729 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-envoy\") pod \"8e1bf62f-188c-4b74-876d-da9212788972\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " Nov 04 01:28:05.478787 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.478759 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1bf62f-188c-4b74-876d-da9212788972-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1bf62f-188c-4b74-876d-da9212788972-istiod-ca-cert\") pod \"8e1bf62f-188c-4b74-876d-da9212788972\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " Nov 04 01:28:05.478841 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.478832 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-istio-token\" (UniqueName: \"kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-istio-token\") pod \"8e1bf62f-188c-4b74-876d-da9212788972\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " Nov 04 01:28:05.478896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.478854 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-socket\") pod \"8e1bf62f-188c-4b74-876d-da9212788972\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " Nov 04 01:28:05.478896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.478860 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-data" pod "8e1bf62f-188c-4b74-876d-da9212788972" (UID: "8e1bf62f-188c-4b74-876d-da9212788972"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:05.479083 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.479030 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-credential-socket" pod "8e1bf62f-188c-4b74-876d-da9212788972" (UID: "8e1bf62f-188c-4b74-876d-da9212788972"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:05.479199 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.479152 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-socket" pod "8e1bf62f-188c-4b74-876d-da9212788972" (UID: "8e1bf62f-188c-4b74-876d-da9212788972"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:05.479262 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.479215 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1bf62f-188c-4b74-876d-da9212788972-istiod-ca-cert" pod "8e1bf62f-188c-4b74-876d-da9212788972" (UID: "8e1bf62f-188c-4b74-876d-da9212788972"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:05.479262 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.479219 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-certs\") pod \"8e1bf62f-188c-4b74-876d-da9212788972\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " Nov 04 01:28:05.479262 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.479223 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-certs" pod "8e1bf62f-188c-4b74-876d-da9212788972" (UID: "8e1bf62f-188c-4b74-876d-da9212788972"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:05.479418 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.479332 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-kube-api-access-ffj2j\" (UniqueName: \"kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-kube-api-access-ffj2j\") pod \"8e1bf62f-188c-4b74-876d-da9212788972\" (UID: \"8e1bf62f-188c-4b74-876d-da9212788972\") " Nov 04 01:28:05.480239 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.480213 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.480355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.480242 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.480355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.480261 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1bf62f-188c-4b74-876d-da9212788972-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.480355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.480276 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.480355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.480294 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.480713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.480689 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e1bf62f-188c-4b74-876d-da9212788972-istio-podinfo" pod "8e1bf62f-188c-4b74-876d-da9212788972" (UID: "8e1bf62f-188c-4b74-876d-da9212788972"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:05.480970 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.480949 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-envoy" pod "8e1bf62f-188c-4b74-876d-da9212788972" (UID: "8e1bf62f-188c-4b74-876d-da9212788972"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:05.481084 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.481069 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-istio-token" pod "8e1bf62f-188c-4b74-876d-da9212788972" (UID: "8e1bf62f-188c-4b74-876d-da9212788972"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:05.481540 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.481523 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-kube-api-access-ffj2j" pod "8e1bf62f-188c-4b74-876d-da9212788972" (UID: "8e1bf62f-188c-4b74-876d-da9212788972"). InnerVolumeSpecName "kube-api-access-ffj2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:05.649007 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.648143 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1bf62f-188c-4b74-876d-da9212788972-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.649007 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.648206 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.649007 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.648273 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffj2j\" (UniqueName: \"kubernetes.io/projected/8e1bf62f-188c-4b74-876d-da9212788972-kube-api-access-ffj2j\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.649007 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.648327 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1bf62f-188c-4b74-876d-da9212788972-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.684903 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.681701 5786 generic.go:356] "Generic (PLEG): container finished" podID="8e1bf62f-188c-4b74-876d-da9212788972" containerID="ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1" exitCode=0 Nov 04 01:28:05.684903 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.681817 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" event={"ID":"8e1bf62f-188c-4b74-876d-da9212788972","Type":"ContainerDied","Data":"ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1"} Nov 04 01:28:05.684903 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.681855 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" event={"ID":"8e1bf62f-188c-4b74-876d-da9212788972","Type":"ContainerDied","Data":"a0bb2eb96a742c916849ee2b57449f5ff2377abf5c1162592f3d57a4e34687cf"} Nov 04 01:28:05.684903 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.681875 5786 scope.go:122] "RemoveContainer" containerID="ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1" Nov 04 01:28:05.684903 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.681968 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn" Nov 04 01:28:05.710439 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.710407 5786 scope.go:122] "RemoveContainer" containerID="d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84" Nov 04 01:28:05.725382 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.725353 5786 scope.go:122] "RemoveContainer" containerID="ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1" Nov 04 01:28:05.725744 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:05.725721 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1\": container with ID starting with ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1 not found: ID does not exist" containerID="ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1" Nov 04 01:28:05.725837 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.725752 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1"} err="failed to get container status \"ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1\": rpc error: code = NotFound desc = could not find container \"ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1\": container with ID starting with ad29c472147f7997502cf71b78e2797d8cb56fb529ee5893c877eec7b6eed8c1 not found: ID does not exist" Nov 04 01:28:05.725837 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.725771 5786 scope.go:122] "RemoveContainer" containerID="d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84" Nov 04 01:28:05.726706 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:05.726676 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84\": container with ID starting with d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84 not found: ID does not exist" containerID="d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84" Nov 04 01:28:05.726795 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.726715 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84"} err="failed to get container status \"d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84\": rpc error: code = NotFound desc = could not find container \"d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84\": container with ID starting with d20e50a1e92173433b2ef0ce6a7b86c906230b0bafcad62a3e4ca9976b56ba84 not found: ID does not exist" Nov 04 01:28:05.854837 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.854800 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn"] Nov 04 01:28:05.858242 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:05.858221 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f10575b3-openshift-default-7889ffd767-4btnn"] Nov 04 01:28:06.057513 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:06.057437 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h"] Nov 04 01:28:06.070784 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:06.070751 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk"] Nov 04 01:28:06.508680 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:06.508650 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1bf62f-188c-4b74-876d-da9212788972" path="/var/lib/kubelet/pods/8e1bf62f-188c-4b74-876d-da9212788972/volumes" Nov 04 01:28:06.509274 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:06.509256 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca5ab91-9b0b-40c0-b7b6-adce26566764" path="/var/lib/kubelet/pods/bca5ab91-9b0b-40c0-b7b6-adce26566764/volumes" Nov 04 01:28:06.745456 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:06.745422 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j"] Nov 04 01:28:06.745681 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:06.745655 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" containerID="cri-o://0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275" gracePeriod=30 Nov 04 01:28:08.852727 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.852693 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj"] Nov 04 01:28:08.853470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853452 5786 cpu_manager.go:414] "containerMap: removing container" podUID="d77462f5-8fc6-460f-9cac-ca3f3ba0526b" containerName="payload-pre-processing" Nov 04 01:28:08.853520 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853471 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77462f5-8fc6-460f-9cac-ca3f3ba0526b" containerName="payload-pre-processing" Nov 04 01:28:08.853520 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853492 5786 cpu_manager.go:414] "containerMap: removing container" podUID="8ca567d6-2994-4217-aab6-0ab526410934" containerName="payload-processing" Nov 04 01:28:08.853520 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853499 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca567d6-2994-4217-aab6-0ab526410934" containerName="payload-processing" Nov 04 01:28:08.853520 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853508 5786 cpu_manager.go:414] "containerMap: removing container" podUID="ade00d43-25af-4c67-bf66-53fd5a3af630" containerName="maas-api" Nov 04 01:28:08.853520 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853514 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade00d43-25af-4c67-bf66-53fd5a3af630" containerName="maas-api" Nov 04 01:28:08.853665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853524 5786 cpu_manager.go:414] "containerMap: removing container" podUID="193ccb6c-79b7-47b9-8614-5fc505f86245" containerName="payload-processing" Nov 04 01:28:08.853665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853529 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193ccb6c-79b7-47b9-8614-5fc505f86245" containerName="payload-processing" Nov 04 01:28:08.853665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853546 5786 cpu_manager.go:414] "containerMap: removing container" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" Nov 04 01:28:08.853665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853550 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" Nov 04 01:28:08.853665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853569 5786 cpu_manager.go:414] "containerMap: removing container" podUID="bca5ab91-9b0b-40c0-b7b6-adce26566764" containerName="maas-api" Nov 04 01:28:08.853665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853574 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca5ab91-9b0b-40c0-b7b6-adce26566764" containerName="maas-api" Nov 04 01:28:08.853665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853580 5786 cpu_manager.go:414] "containerMap: removing container" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" Nov 04 01:28:08.853665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853584 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" Nov 04 01:28:08.853665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853593 5786 cpu_manager.go:414] "containerMap: removing container" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" Nov 04 01:28:08.853665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853598 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" Nov 04 01:28:08.853940 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853699 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca5ab91-9b0b-40c0-b7b6-adce26566764" containerName="maas-api" Nov 04 01:28:08.853940 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853710 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="f457f48b-d5aa-43d0-ab4d-0df141a949c1" containerName="main" Nov 04 01:28:08.853940 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853720 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" Nov 04 01:28:08.853940 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853728 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77462f5-8fc6-460f-9cac-ca3f3ba0526b" containerName="payload-pre-processing" Nov 04 01:28:08.853940 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853735 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca567d6-2994-4217-aab6-0ab526410934" containerName="payload-processing" Nov 04 01:28:08.853940 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853741 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade00d43-25af-4c67-bf66-53fd5a3af630" containerName="maas-api" Nov 04 01:28:08.853940 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853746 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="193ccb6c-79b7-47b9-8614-5fc505f86245" containerName="payload-processing" Nov 04 01:28:08.853940 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.853755 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1bf62f-188c-4b74-876d-da9212788972" containerName="istio-proxy" Nov 04 01:28:08.857892 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.857876 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:08.936111 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.936074 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj"] Nov 04 01:28:08.958174 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.958135 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-5778ad-openshift-default-dockercfg-t7xnd\"" Nov 04 01:28:08.992512 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.992479 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3437edcf-befd-4454-b1db-f3ec59979527-istiod-ca-cert\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:08.992697 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.992524 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-envoy\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:08.992742 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.992704 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-certs\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:08.992788 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.992771 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcn6d\" (UniqueName: \"kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-kube-api-access-pcn6d\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:08.992905 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.992882 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-socket\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:08.992982 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.992924 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-istio-token\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:08.993019 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.992985 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-credential-socket\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:08.993019 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.993007 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-data\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:08.993132 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:08.993104 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3437edcf-befd-4454-b1db-f3ec59979527-istio-podinfo\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.098345 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3437edcf-befd-4454-b1db-f3ec59979527-istiod-ca-cert\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.098454 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-envoy\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.098642 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-certs\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.098731 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcn6d\" (UniqueName: \"kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-kube-api-access-pcn6d\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.098800 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-socket\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.098838 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-istio-token\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.098865 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-credential-socket\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.098894 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-data\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.098976 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3437edcf-befd-4454-b1db-f3ec59979527-istio-podinfo\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.100704 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-socket\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.100964 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-credential-socket\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101875 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.101277 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3437edcf-befd-4454-b1db-f3ec59979527-istiod-ca-cert\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.101953 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.101872 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-certs\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.117192 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.113383 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-data\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.124148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.120717 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3437edcf-befd-4454-b1db-f3ec59979527-istio-podinfo\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.124689 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.124328 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-envoy\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.170632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.170061 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-istio-token\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.172971 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.172932 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcn6d\" (UniqueName: \"kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-kube-api-access-pcn6d\") pod \"e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.205089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.205057 5786 patch_prober.go:29] interesting pod/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:09.205261 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.205160 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:09.472090 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.471700 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:09.903590 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.903560 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj"] Nov 04 01:28:09.907332 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:28:09.907305 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3437edcf_befd_4454_b1db_f3ec59979527.slice/crio-a4318f55ba5aa1abee8c22875132c6ef8542f19f54abd92322cbd9f2bf3abd3c WatchSource:0}: Error finding container a4318f55ba5aa1abee8c22875132c6ef8542f19f54abd92322cbd9f2bf3abd3c: Status 404 returned error can't find the container with id a4318f55ba5aa1abee8c22875132c6ef8542f19f54abd92322cbd9f2bf3abd3c Nov 04 01:28:09.910054 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.910018 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:09.910162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.910089 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:09.910162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:09.910133 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:10.724012 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:10.723976 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" event={"ID":"3437edcf-befd-4454-b1db-f3ec59979527","Type":"ContainerStarted","Data":"0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8"} Nov 04 01:28:10.724223 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:10.724041 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" event={"ID":"3437edcf-befd-4454-b1db-f3ec59979527","Type":"ContainerStarted","Data":"a4318f55ba5aa1abee8c22875132c6ef8542f19f54abd92322cbd9f2bf3abd3c"} Nov 04 01:28:10.943085 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:10.943012 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" podStartSLOduration=2.942991986 podStartE2EDuration="2.942991986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:10.887332033 +0000 UTC m=+1836.503677797" watchObservedRunningTime="2026-11-04 01:28:10.942991986 +0000 UTC m=+1836.559337750" Nov 04 01:28:11.472437 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:11.472402 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:11.478457 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:11.478430 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:11.827311 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:11.827282 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:11.829459 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:11.829433 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:28:12.014735 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.010401 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c"] Nov 04 01:28:12.015496 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.015438 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" Nov 04 01:28:12.066739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.066651 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-3dc6c8\"" Nov 04 01:28:12.067137 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.067099 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-3dc6c8-dockercfg-9cz99\"" Nov 04 01:28:12.090851 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.086886 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c"] Nov 04 01:28:12.180778 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.179491 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5862d90-590b-4eb4-bd40-8ca937bccf4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c\" (UID: \"a5862d90-590b-4eb4-bd40-8ca937bccf4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" Nov 04 01:28:12.180778 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.179660 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttg2r\" (UniqueName: \"kubernetes.io/projected/a5862d90-590b-4eb4-bd40-8ca937bccf4c-kube-api-access-ttg2r\") pod \"payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c\" (UID: \"a5862d90-590b-4eb4-bd40-8ca937bccf4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" Nov 04 01:28:12.187548 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.187517 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc"] Nov 04 01:28:12.198433 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.198402 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" Nov 04 01:28:12.300552 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.300504 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5862d90-590b-4eb4-bd40-8ca937bccf4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c\" (UID: \"a5862d90-590b-4eb4-bd40-8ca937bccf4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" Nov 04 01:28:12.303410 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.303372 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttg2r\" (UniqueName: \"kubernetes.io/projected/a5862d90-590b-4eb4-bd40-8ca937bccf4c-kube-api-access-ttg2r\") pod \"payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c\" (UID: \"a5862d90-590b-4eb4-bd40-8ca937bccf4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" Nov 04 01:28:12.326194 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.305662 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5862d90-590b-4eb4-bd40-8ca937bccf4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c\" (UID: \"a5862d90-590b-4eb4-bd40-8ca937bccf4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" Nov 04 01:28:12.429187 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.414986 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87c612a-502f-4488-b9d1-57f1d4bee521-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc\" (UID: \"f87c612a-502f-4488-b9d1-57f1d4bee521\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" Nov 04 01:28:12.437148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.430541 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482lf\" (UniqueName: \"kubernetes.io/projected/f87c612a-502f-4488-b9d1-57f1d4bee521-kube-api-access-482lf\") pod \"payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc\" (UID: \"f87c612a-502f-4488-b9d1-57f1d4bee521\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" Nov 04 01:28:12.543470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.543423 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482lf\" (UniqueName: \"kubernetes.io/projected/f87c612a-502f-4488-b9d1-57f1d4bee521-kube-api-access-482lf\") pod \"payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc\" (UID: \"f87c612a-502f-4488-b9d1-57f1d4bee521\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" Nov 04 01:28:12.543856 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.543837 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87c612a-502f-4488-b9d1-57f1d4bee521-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc\" (UID: \"f87c612a-502f-4488-b9d1-57f1d4bee521\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" Nov 04 01:28:12.631233 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.630319 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87c612a-502f-4488-b9d1-57f1d4bee521-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc\" (UID: \"f87c612a-502f-4488-b9d1-57f1d4bee521\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" Nov 04 01:28:12.657603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.640604 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-482lf\" (UniqueName: \"kubernetes.io/projected/f87c612a-502f-4488-b9d1-57f1d4bee521-kube-api-access-482lf\") pod \"payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc\" (UID: \"f87c612a-502f-4488-b9d1-57f1d4bee521\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" Nov 04 01:28:12.657808 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.657636 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttg2r\" (UniqueName: \"kubernetes.io/projected/a5862d90-590b-4eb4-bd40-8ca937bccf4c-kube-api-access-ttg2r\") pod \"payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c\" (UID: \"a5862d90-590b-4eb4-bd40-8ca937bccf4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" Nov 04 01:28:12.659162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.659113 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc"] Nov 04 01:28:12.696230 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.695095 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" Nov 04 01:28:12.732169 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.732101 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:28:12.748485 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.748441 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-token\" (UniqueName: \"kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-token\") pod \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " Nov 04 01:28:12.748668 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.748607 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-kube-api-access-rnn5t\" (UniqueName: \"kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-kube-api-access-rnn5t\") pod \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " Nov 04 01:28:12.748747 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.748691 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-data\") pod \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " Nov 04 01:28:12.748747 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.748721 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-credential-socket\") pod \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " Nov 04 01:28:12.748904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.748886 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istiod-ca-cert\") pod \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " Nov 04 01:28:12.748967 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.748933 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-certs\") pod \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " Nov 04 01:28:12.749030 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.748962 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-socket\") pod \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " Nov 04 01:28:12.749030 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.749003 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-podinfo\") pod \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " Nov 04 01:28:12.749157 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.749014 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-data" pod "7181de9e-3d8a-409e-bdd8-de4afd7b4be1" (UID: "7181de9e-3d8a-409e-bdd8-de4afd7b4be1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:12.749157 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.749030 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-credential-socket" pod "7181de9e-3d8a-409e-bdd8-de4afd7b4be1" (UID: "7181de9e-3d8a-409e-bdd8-de4afd7b4be1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:12.749314 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.749290 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-certs" pod "7181de9e-3d8a-409e-bdd8-de4afd7b4be1" (UID: "7181de9e-3d8a-409e-bdd8-de4afd7b4be1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:12.749374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.749325 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istiod-ca-cert" pod "7181de9e-3d8a-409e-bdd8-de4afd7b4be1" (UID: "7181de9e-3d8a-409e-bdd8-de4afd7b4be1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:12.749586 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.749561 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-socket" pod "7181de9e-3d8a-409e-bdd8-de4afd7b4be1" (UID: "7181de9e-3d8a-409e-bdd8-de4afd7b4be1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:12.750896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.750855 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-token" pod "7181de9e-3d8a-409e-bdd8-de4afd7b4be1" (UID: "7181de9e-3d8a-409e-bdd8-de4afd7b4be1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:12.751041 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.750906 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-kube-api-access-rnn5t" pod "7181de9e-3d8a-409e-bdd8-de4afd7b4be1" (UID: "7181de9e-3d8a-409e-bdd8-de4afd7b4be1"). InnerVolumeSpecName "kube-api-access-rnn5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:12.751041 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.751006 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-podinfo" pod "7181de9e-3d8a-409e-bdd8-de4afd7b4be1" (UID: "7181de9e-3d8a-409e-bdd8-de4afd7b4be1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:12.752945 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.752917 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:12.753034 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.752952 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:12.753034 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.752968 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:12.753034 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.752982 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:12.753142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.753034 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:12.753142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.753052 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnn5t\" (UniqueName: \"kubernetes.io/projected/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-kube-api-access-rnn5t\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:12.753142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.753066 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:12.753142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.753082 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:12.794651 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.794356 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" Nov 04 01:28:12.838110 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.838078 5786 generic.go:356] "Generic (PLEG): container finished" podID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerID="0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275" exitCode=0 Nov 04 01:28:12.838980 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.838961 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" Nov 04 01:28:12.839114 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.839095 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" event={"ID":"7181de9e-3d8a-409e-bdd8-de4afd7b4be1","Type":"ContainerDied","Data":"0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275"} Nov 04 01:28:12.839217 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.839152 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j" event={"ID":"7181de9e-3d8a-409e-bdd8-de4afd7b4be1","Type":"ContainerDied","Data":"17dfcead92a79568688d299fbdb2fde4d425304592df05250e2a86a555d49093"} Nov 04 01:28:12.839217 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.839175 5786 scope.go:122] "RemoveContainer" containerID="0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275" Nov 04 01:28:12.851435 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.851412 5786 scope.go:122] "RemoveContainer" containerID="7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf" Nov 04 01:28:12.854222 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.853783 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-envoy\") pod \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\" (UID: \"7181de9e-3d8a-409e-bdd8-de4afd7b4be1\") " Nov 04 01:28:12.858674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.858648 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-envoy" pod "7181de9e-3d8a-409e-bdd8-de4afd7b4be1" (UID: "7181de9e-3d8a-409e-bdd8-de4afd7b4be1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:12.891582 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.891562 5786 scope.go:122] "RemoveContainer" containerID="0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275" Nov 04 01:28:12.892020 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:12.891991 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275\": container with ID starting with 0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275 not found: ID does not exist" containerID="0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275" Nov 04 01:28:12.892106 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.892032 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275"} err="failed to get container status \"0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275\": rpc error: code = NotFound desc = could not find container \"0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275\": container with ID starting with 0977b4405cc0de75109ddd9f9c6a73eddb3af3c775b5025503c94b596d26e275 not found: ID does not exist" Nov 04 01:28:12.892106 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.892058 5786 scope.go:122] "RemoveContainer" containerID="7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf" Nov 04 01:28:12.892472 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:12.892434 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf\": container with ID starting with 7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf not found: ID does not exist" containerID="7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf" Nov 04 01:28:12.892587 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.892482 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf"} err="failed to get container status \"7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf\": rpc error: code = NotFound desc = could not find container \"7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf\": container with ID starting with 7ed70ea5edff6b782a83baea675d9bac9fd3d079a82e36aff5feb489d8884eaf not found: ID does not exist" Nov 04 01:28:12.957147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:12.957089 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7181de9e-3d8a-409e-bdd8-de4afd7b4be1-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.096397 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:13.096362 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c"] Nov 04 01:28:13.100000 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:28:13.099973 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5862d90_590b_4eb4_bd40_8ca937bccf4c.slice/crio-1d0307ca13e96fb3723ab89857809677b016d13fb77b3c62964fb974e4fff9bc WatchSource:0}: Error finding container 1d0307ca13e96fb3723ab89857809677b016d13fb77b3c62964fb974e4fff9bc: Status 404 returned error can't find the container with id 1d0307ca13e96fb3723ab89857809677b016d13fb77b3c62964fb974e4fff9bc Nov 04 01:28:13.199461 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:13.199437 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc"] Nov 04 01:28:13.202646 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:28:13.202617 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87c612a_502f_4488_b9d1_57f1d4bee521.slice/crio-8aa3bf50eee2fce793455d27a9e3dfd3c9c29b58ce4e7c11e43af7d2151783ba WatchSource:0}: Error finding container 8aa3bf50eee2fce793455d27a9e3dfd3c9c29b58ce4e7c11e43af7d2151783ba: Status 404 returned error can't find the container with id 8aa3bf50eee2fce793455d27a9e3dfd3c9c29b58ce4e7c11e43af7d2151783ba Nov 04 01:28:13.286630 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:13.286590 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j"] Nov 04 01:28:13.290554 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:13.290526 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d3d6ea20-openshift-default-79cbd8c675-qmn5j"] Nov 04 01:28:13.853808 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:13.853774 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" event={"ID":"a5862d90-590b-4eb4-bd40-8ca937bccf4c","Type":"ContainerStarted","Data":"3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae"} Nov 04 01:28:13.854007 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:13.853863 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" event={"ID":"a5862d90-590b-4eb4-bd40-8ca937bccf4c","Type":"ContainerStarted","Data":"1d0307ca13e96fb3723ab89857809677b016d13fb77b3c62964fb974e4fff9bc"} Nov 04 01:28:13.854007 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:13.853959 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" Nov 04 01:28:13.856631 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:13.856605 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" event={"ID":"f87c612a-502f-4488-b9d1-57f1d4bee521","Type":"ContainerStarted","Data":"3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff"} Nov 04 01:28:13.856754 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:13.856650 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" event={"ID":"f87c612a-502f-4488-b9d1-57f1d4bee521","Type":"ContainerStarted","Data":"8aa3bf50eee2fce793455d27a9e3dfd3c9c29b58ce4e7c11e43af7d2151783ba"} Nov 04 01:28:13.856754 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:13.856691 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" Nov 04 01:28:13.968081 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:13.968021 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" podStartSLOduration=2.968006552 podStartE2EDuration="2.968006552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:13.966460954 +0000 UTC m=+1839.582812356" watchObservedRunningTime="2026-11-04 01:28:13.968006552 +0000 UTC m=+1839.584352354" Nov 04 01:28:14.082523 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:14.082468 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" podStartSLOduration=2.082446277 podStartE2EDuration="2.082446277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:14.080678882 +0000 UTC m=+1839.697024646" watchObservedRunningTime="2026-11-04 01:28:14.082446277 +0000 UTC m=+1839.698792040" Nov 04 01:28:14.542213 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:14.541402 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" path="/var/lib/kubelet/pods/7181de9e-3d8a-409e-bdd8-de4afd7b4be1/volumes" Nov 04 01:28:15.095502 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:15.095469 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg"] Nov 04 01:28:15.096888 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:15.096858 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:28:15.227483 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:15.227448 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888"] Nov 04 01:28:20.096191 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.096136 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" podUID="b4e57d7b-8a83-465f-a030-d98122b1935f" containerName="payload-processing" containerID="cri-o://1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b" gracePeriod=25 Nov 04 01:28:20.228554 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.228518 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" podUID="e9e7cd47-ec58-4d25-abac-8b3e3001f3a0" containerName="payload-pre-processing" containerID="cri-o://ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025" gracePeriod=25 Nov 04 01:28:20.375309 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.375278 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" Nov 04 01:28:20.471531 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.471497 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e57d7b-8a83-465f-a030-d98122b1935f-kube-api-access-qjwjm\" (UniqueName: \"kubernetes.io/projected/b4e57d7b-8a83-465f-a030-d98122b1935f-kube-api-access-qjwjm\") pod \"b4e57d7b-8a83-465f-a030-d98122b1935f\" (UID: \"b4e57d7b-8a83-465f-a030-d98122b1935f\") " Nov 04 01:28:20.471879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.471856 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e57d7b-8a83-465f-a030-d98122b1935f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e57d7b-8a83-465f-a030-d98122b1935f-plugins-config-volume\") pod \"b4e57d7b-8a83-465f-a030-d98122b1935f\" (UID: \"b4e57d7b-8a83-465f-a030-d98122b1935f\") " Nov 04 01:28:20.472344 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.472318 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e57d7b-8a83-465f-a030-d98122b1935f-plugins-config-volume" pod "b4e57d7b-8a83-465f-a030-d98122b1935f" (UID: "b4e57d7b-8a83-465f-a030-d98122b1935f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:20.472576 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.472556 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e57d7b-8a83-465f-a030-d98122b1935f-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:20.474400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.474371 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e57d7b-8a83-465f-a030-d98122b1935f-kube-api-access-qjwjm" pod "b4e57d7b-8a83-465f-a030-d98122b1935f" (UID: "b4e57d7b-8a83-465f-a030-d98122b1935f"). InnerVolumeSpecName "kube-api-access-qjwjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:20.521512 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.521475 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" Nov 04 01:28:20.574354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.574323 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjwjm\" (UniqueName: \"kubernetes.io/projected/b4e57d7b-8a83-465f-a030-d98122b1935f-kube-api-access-qjwjm\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:20.676105 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.676004 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-kube-api-access-28tq8\" (UniqueName: \"kubernetes.io/projected/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-kube-api-access-28tq8\") pod \"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0\" (UID: \"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0\") " Nov 04 01:28:20.676296 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.676209 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-plugins-config-volume\") pod \"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0\" (UID: \"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0\") " Nov 04 01:28:20.676608 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.676581 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-plugins-config-volume" pod "e9e7cd47-ec58-4d25-abac-8b3e3001f3a0" (UID: "e9e7cd47-ec58-4d25-abac-8b3e3001f3a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:20.677058 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.677033 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:20.678310 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.678288 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-kube-api-access-28tq8" pod "e9e7cd47-ec58-4d25-abac-8b3e3001f3a0" (UID: "e9e7cd47-ec58-4d25-abac-8b3e3001f3a0"). InnerVolumeSpecName "kube-api-access-28tq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:20.778660 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.778628 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28tq8\" (UniqueName: \"kubernetes.io/projected/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0-kube-api-access-28tq8\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:20.987814 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.987723 5786 generic.go:356] "Generic (PLEG): container finished" podID="b4e57d7b-8a83-465f-a030-d98122b1935f" containerID="1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b" exitCode=0 Nov 04 01:28:20.987814 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.987786 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" Nov 04 01:28:20.988050 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.987804 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" event={"ID":"b4e57d7b-8a83-465f-a030-d98122b1935f","Type":"ContainerDied","Data":"1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b"} Nov 04 01:28:20.988050 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.987856 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg" event={"ID":"b4e57d7b-8a83-465f-a030-d98122b1935f","Type":"ContainerDied","Data":"5108d3e486184fa9abfc9f228b6e87b2d34ae9df3bacacb8d7da993dab98c434"} Nov 04 01:28:20.988050 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.987881 5786 scope.go:122] "RemoveContainer" containerID="1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b" Nov 04 01:28:20.989692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.989664 5786 generic.go:356] "Generic (PLEG): container finished" podID="e9e7cd47-ec58-4d25-abac-8b3e3001f3a0" containerID="ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025" exitCode=0 Nov 04 01:28:20.989817 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.989714 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" Nov 04 01:28:20.989817 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.989749 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" event={"ID":"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0","Type":"ContainerDied","Data":"ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025"} Nov 04 01:28:20.989817 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:20.989803 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888" event={"ID":"e9e7cd47-ec58-4d25-abac-8b3e3001f3a0","Type":"ContainerDied","Data":"0575b80ecebbdca1a2a905f85a7ac6fd823c06725e81f383c273c4a166fc828f"} Nov 04 01:28:21.003289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.002629 5786 scope.go:122] "RemoveContainer" containerID="1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b" Nov 04 01:28:21.003289 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:21.003230 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b\": container with ID starting with 1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b not found: ID does not exist" containerID="1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b" Nov 04 01:28:21.003289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.003260 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b"} err="failed to get container status \"1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b\": rpc error: code = NotFound desc = could not find container \"1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b\": container with ID starting with 1249526bc662506a04fdec50c71ef0eb47d885885a9e8e9449dee839ee599e3b not found: ID does not exist" Nov 04 01:28:21.003289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.003284 5786 scope.go:122] "RemoveContainer" containerID="ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025" Nov 04 01:28:21.013065 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.013044 5786 scope.go:122] "RemoveContainer" containerID="ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025" Nov 04 01:28:21.013349 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:21.013329 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025\": container with ID starting with ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025 not found: ID does not exist" containerID="ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025" Nov 04 01:28:21.013430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.013360 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025"} err="failed to get container status \"ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025\": rpc error: code = NotFound desc = could not find container \"ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025\": container with ID starting with ada154121e37e62eff637be2a92ba8f3f493ee5aa1520516405e56eb3b008025 not found: ID does not exist" Nov 04 01:28:21.080193 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.080152 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" containerID="cri-o://adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68" gracePeriod=45 Nov 04 01:28:21.091970 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.091934 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" containerID="cri-o://83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec" gracePeriod=45 Nov 04 01:28:21.153184 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.153152 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg"] Nov 04 01:28:21.199201 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.199173 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c563ee-86c5546698-znjlg"] Nov 04 01:28:21.305048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.305022 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888"] Nov 04 01:28:21.310670 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.310643 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c563ee-647d56ff5-gf888"] Nov 04 01:28:21.393267 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.393238 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:28:21.437484 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.437456 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:28:21.489778 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.489732 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-tmp-dir\") pod \"25b15931-2a05-4727-9d4a-1c589e9be703\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " Nov 04 01:28:21.489988 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.489894 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-home\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-home\") pod \"25b15931-2a05-4727-9d4a-1c589e9be703\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " Nov 04 01:28:21.489988 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.489938 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b15931-2a05-4727-9d4a-1c589e9be703-tls-certs\" (UniqueName: \"kubernetes.io/secret/25b15931-2a05-4727-9d4a-1c589e9be703-tls-certs\") pod \"25b15931-2a05-4727-9d4a-1c589e9be703\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " Nov 04 01:28:21.490109 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.489991 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-model-cache\") pod \"25b15931-2a05-4727-9d4a-1c589e9be703\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " Nov 04 01:28:21.490109 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.490050 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-tmp-dir" pod "25b15931-2a05-4727-9d4a-1c589e9be703" (UID: "25b15931-2a05-4727-9d4a-1c589e9be703"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.490257 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.490063 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-dshm\") pod \"25b15931-2a05-4727-9d4a-1c589e9be703\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " Nov 04 01:28:21.490257 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.490201 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-home" pod "25b15931-2a05-4727-9d4a-1c589e9be703" (UID: "25b15931-2a05-4727-9d4a-1c589e9be703"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.490356 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.490253 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b15931-2a05-4727-9d4a-1c589e9be703-kube-api-access-6gjmd\" (UniqueName: \"kubernetes.io/projected/25b15931-2a05-4727-9d4a-1c589e9be703-kube-api-access-6gjmd\") pod \"25b15931-2a05-4727-9d4a-1c589e9be703\" (UID: \"25b15931-2a05-4727-9d4a-1c589e9be703\") " Nov 04 01:28:21.490404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.490362 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-model-cache" pod "25b15931-2a05-4727-9d4a-1c589e9be703" (UID: "25b15931-2a05-4727-9d4a-1c589e9be703"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.490989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.490962 5786 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-tmp-dir\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.490989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.490989 5786 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-home\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.491212 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.490999 5786 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-model-cache\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.492419 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.492393 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-dshm" pod "25b15931-2a05-4727-9d4a-1c589e9be703" (UID: "25b15931-2a05-4727-9d4a-1c589e9be703"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.492546 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.492524 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b15931-2a05-4727-9d4a-1c589e9be703-tls-certs" pod "25b15931-2a05-4727-9d4a-1c589e9be703" (UID: "25b15931-2a05-4727-9d4a-1c589e9be703"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:21.492546 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.492537 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b15931-2a05-4727-9d4a-1c589e9be703-kube-api-access-6gjmd" pod "25b15931-2a05-4727-9d4a-1c589e9be703" (UID: "25b15931-2a05-4727-9d4a-1c589e9be703"). InnerVolumeSpecName "kube-api-access-6gjmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:21.592437 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.592338 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e80b3fd-636d-4b64-912a-57b1e782dab7-kube-api-access-w429n\" (UniqueName: \"kubernetes.io/projected/5e80b3fd-636d-4b64-912a-57b1e782dab7-kube-api-access-w429n\") pod \"5e80b3fd-636d-4b64-912a-57b1e782dab7\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " Nov 04 01:28:21.592437 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.592415 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-model-cache\") pod \"5e80b3fd-636d-4b64-912a-57b1e782dab7\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " Nov 04 01:28:21.592679 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.592462 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e80b3fd-636d-4b64-912a-57b1e782dab7-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e80b3fd-636d-4b64-912a-57b1e782dab7-tls-certs\") pod \"5e80b3fd-636d-4b64-912a-57b1e782dab7\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " Nov 04 01:28:21.592679 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.592598 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-home\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-home\") pod \"5e80b3fd-636d-4b64-912a-57b1e782dab7\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " Nov 04 01:28:21.592807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.592680 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-model-cache" pod "5e80b3fd-636d-4b64-912a-57b1e782dab7" (UID: "5e80b3fd-636d-4b64-912a-57b1e782dab7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.592807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.592706 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-dshm\") pod \"5e80b3fd-636d-4b64-912a-57b1e782dab7\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " Nov 04 01:28:21.592913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.592805 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-tmp-dir\") pod \"5e80b3fd-636d-4b64-912a-57b1e782dab7\" (UID: \"5e80b3fd-636d-4b64-912a-57b1e782dab7\") " Nov 04 01:28:21.592913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.592844 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-home" pod "5e80b3fd-636d-4b64-912a-57b1e782dab7" (UID: "5e80b3fd-636d-4b64-912a-57b1e782dab7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.593047 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.593009 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-tmp-dir" pod "5e80b3fd-636d-4b64-912a-57b1e782dab7" (UID: "5e80b3fd-636d-4b64-912a-57b1e782dab7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.593593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.593569 5786 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-tmp-dir\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.593706 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.593599 5786 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-model-cache\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.593706 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.593615 5786 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b15931-2a05-4727-9d4a-1c589e9be703-tls-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.593706 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.593629 5786 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b15931-2a05-4727-9d4a-1c589e9be703-dshm\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.593706 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.593644 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gjmd\" (UniqueName: \"kubernetes.io/projected/25b15931-2a05-4727-9d4a-1c589e9be703-kube-api-access-6gjmd\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.593706 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.593659 5786 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-home\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.594652 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.594631 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-dshm" pod "5e80b3fd-636d-4b64-912a-57b1e782dab7" (UID: "5e80b3fd-636d-4b64-912a-57b1e782dab7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.594652 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.594637 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e80b3fd-636d-4b64-912a-57b1e782dab7-tls-certs" pod "5e80b3fd-636d-4b64-912a-57b1e782dab7" (UID: "5e80b3fd-636d-4b64-912a-57b1e782dab7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:21.594652 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.594645 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e80b3fd-636d-4b64-912a-57b1e782dab7-kube-api-access-w429n" pod "5e80b3fd-636d-4b64-912a-57b1e782dab7" (UID: "5e80b3fd-636d-4b64-912a-57b1e782dab7"). InnerVolumeSpecName "kube-api-access-w429n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:21.695151 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.695072 5786 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e80b3fd-636d-4b64-912a-57b1e782dab7-tls-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.695151 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.695133 5786 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e80b3fd-636d-4b64-912a-57b1e782dab7-dshm\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.695151 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.695149 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w429n\" (UniqueName: \"kubernetes.io/projected/5e80b3fd-636d-4b64-912a-57b1e782dab7-kube-api-access-w429n\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.773220 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.773185 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2"] Nov 04 01:28:21.773450 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.773410 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerName="istio-proxy" containerID="cri-o://715fde94ffffffb5dc0fa8788e6031402aee074b997a09b04243b76aeb0ddf40" gracePeriod=30 Nov 04 01:28:21.776437 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.776411 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn"] Nov 04 01:28:21.776703 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:21.776637 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" podUID="061cb628-1307-41db-b238-e432a72f9150" containerName="istio-proxy" containerID="cri-o://be91ab787ebf5234c3000461a552b7a5c76a534a09484a82a992036b3bf3b352" gracePeriod=30 Nov 04 01:28:22.006860 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.006826 5786 generic.go:356] "Generic (PLEG): container finished" podID="25b15931-2a05-4727-9d4a-1c589e9be703" containerID="adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68" exitCode=0 Nov 04 01:28:22.007067 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.006877 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" Nov 04 01:28:22.007067 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.006916 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" event={"ID":"25b15931-2a05-4727-9d4a-1c589e9be703","Type":"ContainerDied","Data":"adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68"} Nov 04 01:28:22.007067 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.006959 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h" event={"ID":"25b15931-2a05-4727-9d4a-1c589e9be703","Type":"ContainerDied","Data":"6e56fbb1c80a309363e0cfa3ca47972af8a1eabd0dbd6df791a07aabcc682892"} Nov 04 01:28:22.007067 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.006981 5786 scope.go:122] "RemoveContainer" containerID="adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68" Nov 04 01:28:22.011390 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.011362 5786 generic.go:356] "Generic (PLEG): container finished" podID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerID="83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec" exitCode=0 Nov 04 01:28:22.011513 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.011423 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" Nov 04 01:28:22.011513 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.011449 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" event={"ID":"5e80b3fd-636d-4b64-912a-57b1e782dab7","Type":"ContainerDied","Data":"83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec"} Nov 04 01:28:22.011513 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.011489 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk" event={"ID":"5e80b3fd-636d-4b64-912a-57b1e782dab7","Type":"ContainerDied","Data":"f474f6887514690c499a6af407c9bef7290c77b446145b78b274755e12d40d13"} Nov 04 01:28:22.019636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.019619 5786 scope.go:122] "RemoveContainer" containerID="adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68" Nov 04 01:28:22.019931 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:22.019909 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68\": container with ID starting with adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68 not found: ID does not exist" containerID="adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68" Nov 04 01:28:22.020015 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.019943 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68"} err="failed to get container status \"adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68\": rpc error: code = NotFound desc = could not find container \"adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68\": container with ID starting with adaa4c3fbf6c316afd55b1a3f5132d5ae9cc33f8a3731062808f71a832e6ea68 not found: ID does not exist" Nov 04 01:28:22.020015 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.019967 5786 scope.go:122] "RemoveContainer" containerID="83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec" Nov 04 01:28:22.030502 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.030482 5786 scope.go:122] "RemoveContainer" containerID="83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec" Nov 04 01:28:22.030799 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:22.030770 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec\": container with ID starting with 83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec not found: ID does not exist" containerID="83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec" Nov 04 01:28:22.030861 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.030801 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec"} err="failed to get container status \"83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec\": rpc error: code = NotFound desc = could not find container \"83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec\": container with ID starting with 83b733a20b3bdb09afdd07938dfb82f3fc3b3a64c66d97c186d12010b62455ec not found: ID does not exist" Nov 04 01:28:22.174676 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.174639 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk"] Nov 04 01:28:22.185257 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.185227 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/simulator-w1-c563ee-kserve-5cf9847c4c-qbgjk"] Nov 04 01:28:22.298685 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.298606 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h"] Nov 04 01:28:22.307767 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.307735 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/premium-w1-c563ee-kserve-77dd759b58-zfs5h"] Nov 04 01:28:22.508415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.508381 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" path="/var/lib/kubelet/pods/25b15931-2a05-4727-9d4a-1c589e9be703/volumes" Nov 04 01:28:22.508873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.508857 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" path="/var/lib/kubelet/pods/5e80b3fd-636d-4b64-912a-57b1e782dab7/volumes" Nov 04 01:28:22.509280 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.509266 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e57d7b-8a83-465f-a030-d98122b1935f" path="/var/lib/kubelet/pods/b4e57d7b-8a83-465f-a030-d98122b1935f/volumes" Nov 04 01:28:22.509965 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.509949 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e7cd47-ec58-4d25-abac-8b3e3001f3a0" path="/var/lib/kubelet/pods/e9e7cd47-ec58-4d25-abac-8b3e3001f3a0/volumes" Nov 04 01:28:22.729816 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.729780 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l"] Nov 04 01:28:22.730068 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:22.730037 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" podUID="ef17f99d-84f8-4f1c-afbe-266df72a8a99" containerName="istio-proxy" containerID="cri-o://5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32" gracePeriod=30 Nov 04 01:28:23.707192 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:23.707144 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:28:23.931377 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:23.931343 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k"] Nov 04 01:28:23.931615 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:23.931586 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerName="istio-proxy" containerID="cri-o://a90e461a8ce7893a989c8e45fa7ff49b6f5dd72b6787fb771d6968096bb1abe4" gracePeriod=30 Nov 04 01:28:23.935167 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:23.935143 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8"] Nov 04 01:28:23.935334 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:23.935314 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerName="istio-proxy" containerID="cri-o://595fdf925fe7979826a2ddeee1f740d49404bbe3d7e688cffb389d64a4cf6903" gracePeriod=30 Nov 04 01:28:23.978667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:23.978560 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2"] Nov 04 01:28:23.978870 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:23.978842 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" containerID="cri-o://fb4d659c33c19af3fdf8cdcc6030405e6700122636e173ee9ab49a72b59e6583" gracePeriod=30 Nov 04 01:28:23.983314 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:23.983283 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw"] Nov 04 01:28:23.983524 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:23.983499 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" containerID="cri-o://d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c" gracePeriod=30 Nov 04 01:28:24.961618 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:24.961582 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" Nov 04 01:28:25.067283 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.067249 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" Nov 04 01:28:25.067525 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.067506 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt"] Nov 04 01:28:25.105141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.068686 5786 cpu_manager.go:414] "containerMap: removing container" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" Nov 04 01:28:25.105415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.105387 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" Nov 04 01:28:25.105565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.105549 5786 cpu_manager.go:414] "containerMap: removing container" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" Nov 04 01:28:25.114143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.105650 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" Nov 04 01:28:25.130214 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.130177 5786 cpu_manager.go:414] "containerMap: removing container" podUID="e9e7cd47-ec58-4d25-abac-8b3e3001f3a0" containerName="payload-pre-processing" Nov 04 01:28:25.159192 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.159156 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e7cd47-ec58-4d25-abac-8b3e3001f3a0" containerName="payload-pre-processing" Nov 04 01:28:25.159458 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.159436 5786 cpu_manager.go:414] "containerMap: removing container" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" Nov 04 01:28:25.159582 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.159567 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" Nov 04 01:28:25.159676 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.159665 5786 cpu_manager.go:414] "containerMap: removing container" podUID="b4e57d7b-8a83-465f-a030-d98122b1935f" containerName="payload-processing" Nov 04 01:28:25.159763 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.159752 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e57d7b-8a83-465f-a030-d98122b1935f" containerName="payload-processing" Nov 04 01:28:25.185143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.160145 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" Nov 04 01:28:25.195220 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.185401 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e80b3fd-636d-4b64-912a-57b1e782dab7" containerName="main" Nov 04 01:28:25.195473 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.195444 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e7cd47-ec58-4d25-abac-8b3e3001f3a0" containerName="payload-pre-processing" Nov 04 01:28:25.195597 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.195586 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b15931-2a05-4727-9d4a-1c589e9be703" containerName="main" Nov 04 01:28:25.201148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.195718 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e57d7b-8a83-465f-a030-d98122b1935f" containerName="payload-processing" Nov 04 01:28:25.316211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.306735 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt"] Nov 04 01:28:25.316211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.306793 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq"] Nov 04 01:28:25.316211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.308346 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" Nov 04 01:28:25.316211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.308449 5786 cpu_manager.go:414] "containerMap: removing container" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" Nov 04 01:28:25.316211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.308470 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" Nov 04 01:28:25.316211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.308920 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="7181de9e-3d8a-409e-bdd8-de4afd7b4be1" containerName="istio-proxy" Nov 04 01:28:25.316211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.314254 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq"] Nov 04 01:28:25.316211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.314375 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" Nov 04 01:28:25.362707 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.361976 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-5778ad-dockercfg-cg8rh\"" Nov 04 01:28:25.362707 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.362208 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-5778ad\"" Nov 04 01:28:25.460189 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.457201 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27xd\" (UniqueName: \"kubernetes.io/projected/fb97732a-38f0-4cf8-a6eb-44a54a99347b-kube-api-access-n27xd\") pod \"payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq\" (UID: \"fb97732a-38f0-4cf8-a6eb-44a54a99347b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" Nov 04 01:28:25.520377 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.503958 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e113e2-7693-402f-a137-f37c551d250e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt\" (UID: \"02e113e2-7693-402f-a137-f37c551d250e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" Nov 04 01:28:25.520377 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.504069 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb97732a-38f0-4cf8-a6eb-44a54a99347b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq\" (UID: \"fb97732a-38f0-4cf8-a6eb-44a54a99347b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" Nov 04 01:28:25.520377 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.504107 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54b2m\" (UniqueName: \"kubernetes.io/projected/02e113e2-7693-402f-a137-f37c551d250e-kube-api-access-54b2m\") pod \"payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt\" (UID: \"02e113e2-7693-402f-a137-f37c551d250e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" Nov 04 01:28:25.662206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.611342 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27xd\" (UniqueName: \"kubernetes.io/projected/fb97732a-38f0-4cf8-a6eb-44a54a99347b-kube-api-access-n27xd\") pod \"payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq\" (UID: \"fb97732a-38f0-4cf8-a6eb-44a54a99347b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" Nov 04 01:28:25.662206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.611466 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e113e2-7693-402f-a137-f37c551d250e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt\" (UID: \"02e113e2-7693-402f-a137-f37c551d250e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" Nov 04 01:28:25.662206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.611520 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb97732a-38f0-4cf8-a6eb-44a54a99347b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq\" (UID: \"fb97732a-38f0-4cf8-a6eb-44a54a99347b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" Nov 04 01:28:25.662206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.611555 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54b2m\" (UniqueName: \"kubernetes.io/projected/02e113e2-7693-402f-a137-f37c551d250e-kube-api-access-54b2m\") pod \"payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt\" (UID: \"02e113e2-7693-402f-a137-f37c551d250e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" Nov 04 01:28:25.663972 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.663935 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e113e2-7693-402f-a137-f37c551d250e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt\" (UID: \"02e113e2-7693-402f-a137-f37c551d250e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" Nov 04 01:28:25.699019 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.695030 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb97732a-38f0-4cf8-a6eb-44a54a99347b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq\" (UID: \"fb97732a-38f0-4cf8-a6eb-44a54a99347b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" Nov 04 01:28:25.757524 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.739534 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54b2m\" (UniqueName: \"kubernetes.io/projected/02e113e2-7693-402f-a137-f37c551d250e-kube-api-access-54b2m\") pod \"payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt\" (UID: \"02e113e2-7693-402f-a137-f37c551d250e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" Nov 04 01:28:25.782223 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.782187 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27xd\" (UniqueName: \"kubernetes.io/projected/fb97732a-38f0-4cf8-a6eb-44a54a99347b-kube-api-access-n27xd\") pod \"payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq\" (UID: \"fb97732a-38f0-4cf8-a6eb-44a54a99347b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" Nov 04 01:28:25.802262 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:25.802217 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" Nov 04 01:28:26.090195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:26.067428 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" Nov 04 01:28:26.250148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:26.250096 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:26.250456 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:26.250426 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:26.313143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:26.268949 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:26.313143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:26.291195 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:26.362656 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:26.362620 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:26.362941 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:26.362914 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:27.147765 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:27.131475 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt"] Nov 04 01:28:27.559761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:27.551861 5786 generic.go:356] "Generic (PLEG): container finished" podID="061cb628-1307-41db-b238-e432a72f9150" containerID="be91ab787ebf5234c3000461a552b7a5c76a534a09484a82a992036b3bf3b352" exitCode=0 Nov 04 01:28:27.559761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:27.552018 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" event={"ID":"061cb628-1307-41db-b238-e432a72f9150","Type":"ContainerDied","Data":"be91ab787ebf5234c3000461a552b7a5c76a534a09484a82a992036b3bf3b352"} Nov 04 01:28:27.559761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:27.555298 5786 generic.go:356] "Generic (PLEG): container finished" podID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerID="715fde94ffffffb5dc0fa8788e6031402aee074b997a09b04243b76aeb0ddf40" exitCode=0 Nov 04 01:28:27.559761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:27.555544 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" event={"ID":"4df90e93-d322-4d8e-87b3-74c38bd2e428","Type":"ContainerDied","Data":"715fde94ffffffb5dc0fa8788e6031402aee074b997a09b04243b76aeb0ddf40"} Nov 04 01:28:27.559761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:27.558009 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" event={"ID":"02e113e2-7693-402f-a137-f37c551d250e","Type":"ContainerStarted","Data":"73ea5d8ffea6a29479c7e4d65b70b1a808a8320ae37a5b11467c54d2b7a00378"} Nov 04 01:28:27.984439 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:27.984348 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb"] Nov 04 01:28:27.984817 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:27.984698 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" podUID="72c067af-133f-494e-b755-926e1047f874" containerName="maas-api" containerID="cri-o://0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1" gracePeriod=30 Nov 04 01:28:28.793083 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.792761 5786 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:28.793083 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.792816 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:28.848494 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.848168 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:28:28.962259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.961702 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/061cb628-1307-41db-b238-e432a72f9150-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061cb628-1307-41db-b238-e432a72f9150-istio-podinfo\") pod \"061cb628-1307-41db-b238-e432a72f9150\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " Nov 04 01:28:28.962259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.961753 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-credential-socket\") pod \"061cb628-1307-41db-b238-e432a72f9150\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " Nov 04 01:28:28.962259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.961817 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061cb628-1307-41db-b238-e432a72f9150-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061cb628-1307-41db-b238-e432a72f9150-istiod-ca-cert\") pod \"061cb628-1307-41db-b238-e432a72f9150\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " Nov 04 01:28:28.962259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.961861 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-data\") pod \"061cb628-1307-41db-b238-e432a72f9150\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " Nov 04 01:28:28.962259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.961917 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-envoy\") pod \"061cb628-1307-41db-b238-e432a72f9150\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " Nov 04 01:28:28.962259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.962037 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-istio-token\" (UniqueName: \"kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-istio-token\") pod \"061cb628-1307-41db-b238-e432a72f9150\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " Nov 04 01:28:28.962692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.962579 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-kube-api-access-dmb98\" (UniqueName: \"kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-kube-api-access-dmb98\") pod \"061cb628-1307-41db-b238-e432a72f9150\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " Nov 04 01:28:28.963437 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.962981 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-certs\") pod \"061cb628-1307-41db-b238-e432a72f9150\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " Nov 04 01:28:28.963437 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.963031 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-socket\") pod \"061cb628-1307-41db-b238-e432a72f9150\" (UID: \"061cb628-1307-41db-b238-e432a72f9150\") " Nov 04 01:28:28.978933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.968366 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-socket" pod "061cb628-1307-41db-b238-e432a72f9150" (UID: "061cb628-1307-41db-b238-e432a72f9150"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.978933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.974299 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-certs" pod "061cb628-1307-41db-b238-e432a72f9150" (UID: "061cb628-1307-41db-b238-e432a72f9150"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.978933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.974612 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-credential-socket" pod "061cb628-1307-41db-b238-e432a72f9150" (UID: "061cb628-1307-41db-b238-e432a72f9150"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.978933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.976945 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061cb628-1307-41db-b238-e432a72f9150-istiod-ca-cert" pod "061cb628-1307-41db-b238-e432a72f9150" (UID: "061cb628-1307-41db-b238-e432a72f9150"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:28.978933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.977339 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-data" pod "061cb628-1307-41db-b238-e432a72f9150" (UID: "061cb628-1307-41db-b238-e432a72f9150"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.999892 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.992267 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-istio-token" pod "061cb628-1307-41db-b238-e432a72f9150" (UID: "061cb628-1307-41db-b238-e432a72f9150"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:28.999892 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.992377 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-envoy" pod "061cb628-1307-41db-b238-e432a72f9150" (UID: "061cb628-1307-41db-b238-e432a72f9150"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.999892 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:28.992434 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/061cb628-1307-41db-b238-e432a72f9150-istio-podinfo" pod "061cb628-1307-41db-b238-e432a72f9150" (UID: "061cb628-1307-41db-b238-e432a72f9150"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:29.002036 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.001995 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-kube-api-access-dmb98" pod "061cb628-1307-41db-b238-e432a72f9150" (UID: "061cb628-1307-41db-b238-e432a72f9150"). InnerVolumeSpecName "kube-api-access-dmb98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:29.259281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.250372 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061cb628-1307-41db-b238-e432a72f9150-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.259281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.250423 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.259281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.250438 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061cb628-1307-41db-b238-e432a72f9150-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.259281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.250450 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.259281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.250462 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.259281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.250481 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.259281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.250496 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmb98\" (UniqueName: \"kubernetes.io/projected/061cb628-1307-41db-b238-e432a72f9150-kube-api-access-dmb98\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.259281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.250509 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.259281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.250525 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061cb628-1307-41db-b238-e432a72f9150-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.410263 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.407966 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq"] Nov 04 01:28:29.437087 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.436994 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:28:29.550454 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.550417 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:28:29.589879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.577475 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-data\") pod \"4df90e93-d322-4d8e-87b3-74c38bd2e428\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " Nov 04 01:28:29.589879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.577550 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-podinfo\") pod \"4df90e93-d322-4d8e-87b3-74c38bd2e428\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " Nov 04 01:28:29.589879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.577597 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-token\" (UniqueName: \"kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-token\") pod \"4df90e93-d322-4d8e-87b3-74c38bd2e428\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " Nov 04 01:28:29.589879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.577687 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-certs\") pod \"4df90e93-d322-4d8e-87b3-74c38bd2e428\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " Nov 04 01:28:29.589879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.577902 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-socket\") pod \"4df90e93-d322-4d8e-87b3-74c38bd2e428\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " Nov 04 01:28:29.589879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.577941 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-envoy\") pod \"4df90e93-d322-4d8e-87b3-74c38bd2e428\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " Nov 04 01:28:29.589879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.578082 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-kube-api-access-gtrbx\" (UniqueName: \"kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-kube-api-access-gtrbx\") pod \"4df90e93-d322-4d8e-87b3-74c38bd2e428\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " Nov 04 01:28:29.589879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.578162 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df90e93-d322-4d8e-87b3-74c38bd2e428-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df90e93-d322-4d8e-87b3-74c38bd2e428-istiod-ca-cert\") pod \"4df90e93-d322-4d8e-87b3-74c38bd2e428\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " Nov 04 01:28:29.589879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.578221 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-credential-socket\") pod \"4df90e93-d322-4d8e-87b3-74c38bd2e428\" (UID: \"4df90e93-d322-4d8e-87b3-74c38bd2e428\") " Nov 04 01:28:29.589879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.586792 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-data" pod "4df90e93-d322-4d8e-87b3-74c38bd2e428" (UID: "4df90e93-d322-4d8e-87b3-74c38bd2e428"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:29.591830 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.591491 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-credential-socket" pod "4df90e93-d322-4d8e-87b3-74c38bd2e428" (UID: "4df90e93-d322-4d8e-87b3-74c38bd2e428"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:29.591830 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.591628 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-token" pod "4df90e93-d322-4d8e-87b3-74c38bd2e428" (UID: "4df90e93-d322-4d8e-87b3-74c38bd2e428"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:29.598895 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.592057 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-socket" pod "4df90e93-d322-4d8e-87b3-74c38bd2e428" (UID: "4df90e93-d322-4d8e-87b3-74c38bd2e428"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:29.598895 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.594202 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-certs" pod "4df90e93-d322-4d8e-87b3-74c38bd2e428" (UID: "4df90e93-d322-4d8e-87b3-74c38bd2e428"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:29.599298 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.599271 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df90e93-d322-4d8e-87b3-74c38bd2e428-istiod-ca-cert" pod "4df90e93-d322-4d8e-87b3-74c38bd2e428" (UID: "4df90e93-d322-4d8e-87b3-74c38bd2e428"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:29.601826 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.601770 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-envoy" pod "4df90e93-d322-4d8e-87b3-74c38bd2e428" (UID: "4df90e93-d322-4d8e-87b3-74c38bd2e428"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:29.604918 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.604885 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-kube-api-access-gtrbx" pod "4df90e93-d322-4d8e-87b3-74c38bd2e428" (UID: "4df90e93-d322-4d8e-87b3-74c38bd2e428"). InnerVolumeSpecName "kube-api-access-gtrbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:29.605017 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.604977 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-podinfo" pod "4df90e93-d322-4d8e-87b3-74c38bd2e428" (UID: "4df90e93-d322-4d8e-87b3-74c38bd2e428"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:29.662205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.661963 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq"] Nov 04 01:28:29.670070 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.670028 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:28:29.680087 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.680051 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c067af-133f-494e-b755-926e1047f874-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c067af-133f-494e-b755-926e1047f874-openshift-service-ca\") pod \"72c067af-133f-494e-b755-926e1047f874\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " Nov 04 01:28:29.680256 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.680201 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-metrics-certs\") pod \"72c067af-133f-494e-b755-926e1047f874\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " Nov 04 01:28:29.680695 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.680651 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c067af-133f-494e-b755-926e1047f874-openshift-service-ca" pod "72c067af-133f-494e-b755-926e1047f874" (UID: "72c067af-133f-494e-b755-926e1047f874"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:29.680806 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.680743 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-maas-api-tls\") pod \"72c067af-133f-494e-b755-926e1047f874\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " Nov 04 01:28:29.680864 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.680822 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c067af-133f-494e-b755-926e1047f874-kube-api-access-k2ww7\" (UniqueName: \"kubernetes.io/projected/72c067af-133f-494e-b755-926e1047f874-kube-api-access-k2ww7\") pod \"72c067af-133f-494e-b755-926e1047f874\" (UID: \"72c067af-133f-494e-b755-926e1047f874\") " Nov 04 01:28:29.681622 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.681596 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c067af-133f-494e-b755-926e1047f874-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.681727 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.681629 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.681727 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.681643 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.681727 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.681658 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtrbx\" (UniqueName: \"kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-kube-api-access-gtrbx\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.681727 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.681672 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df90e93-d322-4d8e-87b3-74c38bd2e428-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.681727 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.681685 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.681727 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.681698 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.681727 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.681711 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.681727 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.681725 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df90e93-d322-4d8e-87b3-74c38bd2e428-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.682108 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.681739 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df90e93-d322-4d8e-87b3-74c38bd2e428-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.703258 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.700783 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c067af-133f-494e-b755-926e1047f874-kube-api-access-k2ww7" pod "72c067af-133f-494e-b755-926e1047f874" (UID: "72c067af-133f-494e-b755-926e1047f874"). InnerVolumeSpecName "kube-api-access-k2ww7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:29.703258 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.701250 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-maas-api-tls" pod "72c067af-133f-494e-b755-926e1047f874" (UID: "72c067af-133f-494e-b755-926e1047f874"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:29.708561 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.708377 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-metrics-certs" pod "72c067af-133f-494e-b755-926e1047f874" (UID: "72c067af-133f-494e-b755-926e1047f874"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:29.765589 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.765553 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl"] Nov 04 01:28:29.782765 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.782730 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-token\" (UniqueName: \"kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-token\") pod \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " Nov 04 01:28:29.782921 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.782871 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istiod-ca-cert\") pod \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " Nov 04 01:28:29.782982 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.782937 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-podinfo\") pod \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " Nov 04 01:28:29.783035 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.783017 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-certs\") pod \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " Nov 04 01:28:29.783087 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.783045 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-socket\") pod \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " Nov 04 01:28:29.783173 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.783111 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-credential-socket\") pod \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " Nov 04 01:28:29.783243 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.783201 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-data\") pod \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " Nov 04 01:28:29.783298 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.783259 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-kube-api-access-8tkhz\" (UniqueName: \"kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-kube-api-access-8tkhz\") pod \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " Nov 04 01:28:29.783389 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.783365 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-certs" pod "ef17f99d-84f8-4f1c-afbe-266df72a8a99" (UID: "ef17f99d-84f8-4f1c-afbe-266df72a8a99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:29.783445 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.783289 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istiod-ca-cert" pod "ef17f99d-84f8-4f1c-afbe-266df72a8a99" (UID: "ef17f99d-84f8-4f1c-afbe-266df72a8a99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:29.783587 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.783549 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-data" pod "ef17f99d-84f8-4f1c-afbe-266df72a8a99" (UID: "ef17f99d-84f8-4f1c-afbe-266df72a8a99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:29.783587 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.783561 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-envoy\") pod \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\" (UID: \"ef17f99d-84f8-4f1c-afbe-266df72a8a99\") " Nov 04 01:28:29.786901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.783677 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-credential-socket" pod "ef17f99d-84f8-4f1c-afbe-266df72a8a99" (UID: "ef17f99d-84f8-4f1c-afbe-266df72a8a99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:29.786901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.783989 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-socket" pod "ef17f99d-84f8-4f1c-afbe-266df72a8a99" (UID: "ef17f99d-84f8-4f1c-afbe-266df72a8a99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:29.786901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.784395 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.786901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.784415 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.786901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.784429 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.786901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.784444 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.786901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.784458 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.786901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.784470 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.786901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.784485 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c067af-133f-494e-b755-926e1047f874-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.786901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.784501 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2ww7\" (UniqueName: \"kubernetes.io/projected/72c067af-133f-494e-b755-926e1047f874-kube-api-access-k2ww7\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.787595 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.787567 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-token" pod "ef17f99d-84f8-4f1c-afbe-266df72a8a99" (UID: "ef17f99d-84f8-4f1c-afbe-266df72a8a99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:29.788981 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.788958 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-envoy" pod "ef17f99d-84f8-4f1c-afbe-266df72a8a99" (UID: "ef17f99d-84f8-4f1c-afbe-266df72a8a99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:29.789071 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.788994 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-kube-api-access-8tkhz" pod "ef17f99d-84f8-4f1c-afbe-266df72a8a99" (UID: "ef17f99d-84f8-4f1c-afbe-266df72a8a99"). InnerVolumeSpecName "kube-api-access-8tkhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:29.790027 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.790001 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-podinfo" pod "ef17f99d-84f8-4f1c-afbe-266df72a8a99" (UID: "ef17f99d-84f8-4f1c-afbe-266df72a8a99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:29.854818 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.854735 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:28:29.856190 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.856155 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" event={"ID":"02e113e2-7693-402f-a137-f37c551d250e","Type":"ContainerStarted","Data":"e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad"} Nov 04 01:28:29.856314 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.856237 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" Nov 04 01:28:29.857225 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.857112 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" event={"ID":"fb97732a-38f0-4cf8-a6eb-44a54a99347b","Type":"ContainerStarted","Data":"36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3"} Nov 04 01:28:29.857389 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.857196 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" Nov 04 01:28:29.857589 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.857547 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" event={"ID":"fb97732a-38f0-4cf8-a6eb-44a54a99347b","Type":"ContainerStarted","Data":"06808acd9e7f0b808c72a22519cd630a23d5e44d1bb670f371e317bfe126b307"} Nov 04 01:28:29.859564 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.859540 5786 generic.go:356] "Generic (PLEG): container finished" podID="ef17f99d-84f8-4f1c-afbe-266df72a8a99" containerID="5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32" exitCode=0 Nov 04 01:28:29.859701 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.859620 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" event={"ID":"ef17f99d-84f8-4f1c-afbe-266df72a8a99","Type":"ContainerDied","Data":"5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32"} Nov 04 01:28:29.859701 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.859667 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" event={"ID":"ef17f99d-84f8-4f1c-afbe-266df72a8a99","Type":"ContainerDied","Data":"6f2333e3a193cbe2e050b5e9402548c8844d2dc3d69541cad074ccf8ba0bb366"} Nov 04 01:28:29.859701 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.859693 5786 scope.go:122] "RemoveContainer" containerID="5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32" Nov 04 01:28:29.860464 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.859919 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l" Nov 04 01:28:29.862303 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.862276 5786 generic.go:356] "Generic (PLEG): container finished" podID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerID="595fdf925fe7979826a2ddeee1f740d49404bbe3d7e688cffb389d64a4cf6903" exitCode=0 Nov 04 01:28:29.862414 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.862395 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" event={"ID":"b9c26c24-a4ba-4ec2-9b68-e38872fa1726","Type":"ContainerDied","Data":"595fdf925fe7979826a2ddeee1f740d49404bbe3d7e688cffb389d64a4cf6903"} Nov 04 01:28:29.864643 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.864617 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" event={"ID":"061cb628-1307-41db-b238-e432a72f9150","Type":"ContainerDied","Data":"533570ec79dac6fadd1379e24fc3d80f7f36adf10dcf23b130f746420137f6f8"} Nov 04 01:28:29.864746 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.864726 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn" Nov 04 01:28:29.872410 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.871786 5786 generic.go:356] "Generic (PLEG): container finished" podID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerID="a90e461a8ce7893a989c8e45fa7ff49b6f5dd72b6787fb771d6968096bb1abe4" exitCode=0 Nov 04 01:28:29.872410 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.871879 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" event={"ID":"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6","Type":"ContainerDied","Data":"a90e461a8ce7893a989c8e45fa7ff49b6f5dd72b6787fb771d6968096bb1abe4"} Nov 04 01:28:29.875752 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.873425 5786 generic.go:356] "Generic (PLEG): container finished" podID="72c067af-133f-494e-b755-926e1047f874" containerID="0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1" exitCode=0 Nov 04 01:28:29.875752 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.873461 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" Nov 04 01:28:29.875752 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.873502 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" event={"ID":"72c067af-133f-494e-b755-926e1047f874","Type":"ContainerDied","Data":"0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1"} Nov 04 01:28:29.875752 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.873549 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb" event={"ID":"72c067af-133f-494e-b755-926e1047f874","Type":"ContainerDied","Data":"4839482bc2c64cab34b6925ddd69b0e9a569dc3c6a886dea5c033c66312a9070"} Nov 04 01:28:29.876323 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.875939 5786 generic.go:356] "Generic (PLEG): container finished" podID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerID="fb4d659c33c19af3fdf8cdcc6030405e6700122636e173ee9ab49a72b59e6583" exitCode=0 Nov 04 01:28:29.876323 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.875995 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" event={"ID":"699f83a5-7fa4-42be-b125-c11e085ca71e","Type":"ContainerDied","Data":"fb4d659c33c19af3fdf8cdcc6030405e6700122636e173ee9ab49a72b59e6583"} Nov 04 01:28:29.878837 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.877853 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" event={"ID":"4df90e93-d322-4d8e-87b3-74c38bd2e428","Type":"ContainerDied","Data":"85bbce46381effe59c7ce42990d33f1c1f61689d503528d0afae23c342bbb323"} Nov 04 01:28:29.878837 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.877964 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" Nov 04 01:28:29.885897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.883277 5786 scope.go:122] "RemoveContainer" containerID="5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32" Nov 04 01:28:29.885897 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:29.885572 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32\": container with ID starting with 5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32 not found: ID does not exist" containerID="5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32" Nov 04 01:28:29.885897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.885606 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32"} err="failed to get container status \"5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32\": rpc error: code = NotFound desc = could not find container \"5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32\": container with ID starting with 5d247e0cb452c3ee4410126ac107e7b315d43243228c28286994843314ce0c32 not found: ID does not exist" Nov 04 01:28:29.885897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.885631 5786 scope.go:122] "RemoveContainer" containerID="be91ab787ebf5234c3000461a552b7a5c76a534a09484a82a992036b3bf3b352" Nov 04 01:28:29.885897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.885811 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.885897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.885834 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tkhz\" (UniqueName: \"kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-kube-api-access-8tkhz\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.885897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.885849 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.885897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.885867 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef17f99d-84f8-4f1c-afbe-266df72a8a99-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:29.888158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.888000 5786 generic.go:356] "Generic (PLEG): container finished" podID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerID="d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c" exitCode=0 Nov 04 01:28:29.888158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.888065 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" event={"ID":"156e3df3-cacc-4b41-9bf5-b2f2d481f57b","Type":"ContainerDied","Data":"d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c"} Nov 04 01:28:29.888158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.888105 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" event={"ID":"156e3df3-cacc-4b41-9bf5-b2f2d481f57b","Type":"ContainerDied","Data":"28ed4541ef2881a4e78c23656cbb7812add37b24eb62aa3a0107321a0c0492be"} Nov 04 01:28:29.888403 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.888176 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw" Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.989800 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-certs\") pod \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.989876 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istiod-ca-cert\") pod \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.989923 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-data\") pod \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.989984 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-socket\") pod \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.990036 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-token\" (UniqueName: \"kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-token\") pod \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.990146 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-podinfo\") pod \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.990193 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-envoy\") pod \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.990289 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-kube-api-access-7hpjc\" (UniqueName: \"kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-kube-api-access-7hpjc\") pod \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.990331 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-credential-socket\") pod \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\" (UID: \"156e3df3-cacc-4b41-9bf5-b2f2d481f57b\") " Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.990413 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-data" pod "156e3df3-cacc-4b41-9bf5-b2f2d481f57b" (UID: "156e3df3-cacc-4b41-9bf5-b2f2d481f57b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.990581 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-certs" pod "156e3df3-cacc-4b41-9bf5-b2f2d481f57b" (UID: "156e3df3-cacc-4b41-9bf5-b2f2d481f57b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.991211 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.991234 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.002446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:29.992044 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istiod-ca-cert" pod "156e3df3-cacc-4b41-9bf5-b2f2d481f57b" (UID: "156e3df3-cacc-4b41-9bf5-b2f2d481f57b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:30.009149 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.006771 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-podinfo" pod "156e3df3-cacc-4b41-9bf5-b2f2d481f57b" (UID: "156e3df3-cacc-4b41-9bf5-b2f2d481f57b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:30.009723 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.009380 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-socket" pod "156e3df3-cacc-4b41-9bf5-b2f2d481f57b" (UID: "156e3df3-cacc-4b41-9bf5-b2f2d481f57b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.030667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.030601 5786 scope.go:122] "RemoveContainer" containerID="0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1" Nov 04 01:28:30.034967 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.034881 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:28:30.038684 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.037523 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-token" pod "156e3df3-cacc-4b41-9bf5-b2f2d481f57b" (UID: "156e3df3-cacc-4b41-9bf5-b2f2d481f57b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:30.038684 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.037620 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-credential-socket" pod "156e3df3-cacc-4b41-9bf5-b2f2d481f57b" (UID: "156e3df3-cacc-4b41-9bf5-b2f2d481f57b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.046033 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.043481 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-kube-api-access-7hpjc" pod "156e3df3-cacc-4b41-9bf5-b2f2d481f57b" (UID: "156e3df3-cacc-4b41-9bf5-b2f2d481f57b"). InnerVolumeSpecName "kube-api-access-7hpjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:30.049046 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.048734 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" podStartSLOduration=6.048718141 podStartE2EDuration="6.048718141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:30.045720083 +0000 UTC m=+1855.662065847" watchObservedRunningTime="2026-11-04 01:28:30.048718141 +0000 UTC m=+1855.665063905" Nov 04 01:28:30.053709 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.053597 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:28:30.054844 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.054792 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-envoy" pod "156e3df3-cacc-4b41-9bf5-b2f2d481f57b" (UID: "156e3df3-cacc-4b41-9bf5-b2f2d481f57b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.071083 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.066357 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:28:30.071083 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.068474 5786 scope.go:122] "RemoveContainer" containerID="0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1" Nov 04 01:28:30.071083 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:30.068758 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1\": container with ID starting with 0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1 not found: ID does not exist" containerID="0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1" Nov 04 01:28:30.071083 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.068788 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1"} err="failed to get container status \"0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1\": rpc error: code = NotFound desc = could not find container \"0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1\": container with ID starting with 0a69b6d0348cd540b3f2f57d076ac8b65b3406d142af8730d788ea22744996a1 not found: ID does not exist" Nov 04 01:28:30.071083 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.068813 5786 scope.go:122] "RemoveContainer" containerID="715fde94ffffffb5dc0fa8788e6031402aee074b997a09b04243b76aeb0ddf40" Nov 04 01:28:30.082654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.082613 5786 scope.go:122] "RemoveContainer" containerID="d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c" Nov 04 01:28:30.093828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.092094 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-data\") pod \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " Nov 04 01:28:30.093828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.092188 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istiod-ca-cert\") pod \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " Nov 04 01:28:30.093828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.092249 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-certs\") pod \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " Nov 04 01:28:30.093828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.092315 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-certs\") pod \"699f83a5-7fa4-42be-b125-c11e085ca71e\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " Nov 04 01:28:30.093828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.092349 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699f83a5-7fa4-42be-b125-c11e085ca71e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699f83a5-7fa4-42be-b125-c11e085ca71e-istiod-ca-cert\") pod \"699f83a5-7fa4-42be-b125-c11e085ca71e\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " Nov 04 01:28:30.093828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.092388 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-podinfo\") pod \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " Nov 04 01:28:30.093828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.092422 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-podinfo\") pod \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " Nov 04 01:28:30.093828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.093077 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699f83a5-7fa4-42be-b125-c11e085ca71e-istiod-ca-cert" pod "699f83a5-7fa4-42be-b125-c11e085ca71e" (UID: "699f83a5-7fa4-42be-b125-c11e085ca71e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:30.093828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.093224 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-data" pod "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" (UID: "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.093828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.093402 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-certs" pod "699f83a5-7fa4-42be-b125-c11e085ca71e" (UID: "699f83a5-7fa4-42be-b125-c11e085ca71e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.093828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.093728 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istiod-ca-cert" pod "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" (UID: "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:30.093828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.093780 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-certs" pod "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" (UID: "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.094736 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.094696 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-credential-socket\") pod \"699f83a5-7fa4-42be-b125-c11e085ca71e\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " Nov 04 01:28:30.094906 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.094865 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-data\") pod \"699f83a5-7fa4-42be-b125-c11e085ca71e\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " Nov 04 01:28:30.095386 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.095015 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-data\") pod \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " Nov 04 01:28:30.095386 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.095205 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-credential-socket" pod "699f83a5-7fa4-42be-b125-c11e085ca71e" (UID: "699f83a5-7fa4-42be-b125-c11e085ca71e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.095386 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.095346 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-data" pod "699f83a5-7fa4-42be-b125-c11e085ca71e" (UID: "699f83a5-7fa4-42be-b125-c11e085ca71e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.095639 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.095512 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-data" pod "b9c26c24-a4ba-4ec2-9b68-e38872fa1726" (UID: "b9c26c24-a4ba-4ec2-9b68-e38872fa1726"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.095854 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.095767 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-envoy\") pod \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " Nov 04 01:28:30.095854 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.095812 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-podinfo" pod "b9c26c24-a4ba-4ec2-9b68-e38872fa1726" (UID: "b9c26c24-a4ba-4ec2-9b68-e38872fa1726"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:30.095854 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.095824 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-kube-api-access-kd5wf\" (UniqueName: \"kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-kube-api-access-kd5wf\") pod \"699f83a5-7fa4-42be-b125-c11e085ca71e\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " Nov 04 01:28:30.095854 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.095852 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-token\" (UniqueName: \"kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-token\") pod \"699f83a5-7fa4-42be-b125-c11e085ca71e\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " Nov 04 01:28:30.096076 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.095941 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-socket\") pod \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " Nov 04 01:28:30.096076 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.095990 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istiod-ca-cert\") pod \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " Nov 04 01:28:30.096076 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.096019 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-certs\") pod \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " Nov 04 01:28:30.096076 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.096052 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-credential-socket\") pod \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " Nov 04 01:28:30.096286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.096153 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-socket\") pod \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " Nov 04 01:28:30.096286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.096194 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-kube-api-access-fq4z4\" (UniqueName: \"kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-kube-api-access-fq4z4\") pod \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " Nov 04 01:28:30.096286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.096240 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-token\" (UniqueName: \"kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-token\") pod \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " Nov 04 01:28:30.096446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.096290 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-envoy\") pod \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\" (UID: \"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6\") " Nov 04 01:28:30.096446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.096345 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-kube-api-access-ptvdn\" (UniqueName: \"kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-kube-api-access-ptvdn\") pod \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " Nov 04 01:28:30.096446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.096375 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-envoy\") pod \"699f83a5-7fa4-42be-b125-c11e085ca71e\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " Nov 04 01:28:30.096446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.096412 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-credential-socket\") pod \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " Nov 04 01:28:30.096642 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.096456 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-socket\") pod \"699f83a5-7fa4-42be-b125-c11e085ca71e\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " Nov 04 01:28:30.096642 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.096495 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-podinfo\") pod \"699f83a5-7fa4-42be-b125-c11e085ca71e\" (UID: \"699f83a5-7fa4-42be-b125-c11e085ca71e\") " Nov 04 01:28:30.096642 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.096552 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-token\" (UniqueName: \"kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-token\") pod \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\" (UID: \"b9c26c24-a4ba-4ec2-9b68-e38872fa1726\") " Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.097536 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istiod-ca-cert" pod "b9c26c24-a4ba-4ec2-9b68-e38872fa1726" (UID: "b9c26c24-a4ba-4ec2-9b68-e38872fa1726"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098064 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-credential-socket" pod "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" (UID: "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098073 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098133 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098153 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098170 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098184 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098197 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/699f83a5-7fa4-42be-b125-c11e085ca71e-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098210 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098224 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098236 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098366 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-podinfo" pod "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" (UID: "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098908 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-envoy" pod "b9c26c24-a4ba-4ec2-9b68-e38872fa1726" (UID: "b9c26c24-a4ba-4ec2-9b68-e38872fa1726"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.099032 5786 scope.go:122] "RemoveContainer" containerID="d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.099372 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-credential-socket" pod "b9c26c24-a4ba-4ec2-9b68-e38872fa1726" (UID: "b9c26c24-a4ba-4ec2-9b68-e38872fa1726"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.099412 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-certs" pod "b9c26c24-a4ba-4ec2-9b68-e38872fa1726" (UID: "b9c26c24-a4ba-4ec2-9b68-e38872fa1726"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:30.099504 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c\": container with ID starting with d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c not found: ID does not exist" containerID="d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.099538 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c"} err="failed to get container status \"d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c\": rpc error: code = NotFound desc = could not find container \"d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c\": container with ID starting with d199eb0c44a2d339e09455c24a6b4d860ccc59434b3f4bc74a4910182d46a92c not found: ID does not exist" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.098248 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.099650 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.099662 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.099673 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hpjc\" (UniqueName: \"kubernetes.io/projected/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-kube-api-access-7hpjc\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.099693 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.099707 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.099716 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.099725 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156e3df3-cacc-4b41-9bf5-b2f2d481f57b-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.100371 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-socket" pod "699f83a5-7fa4-42be-b125-c11e085ca71e" (UID: "699f83a5-7fa4-42be-b125-c11e085ca71e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.100640 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.100381 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-socket" pod "b9c26c24-a4ba-4ec2-9b68-e38872fa1726" (UID: "b9c26c24-a4ba-4ec2-9b68-e38872fa1726"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.102300 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.100874 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-socket" pod "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" (UID: "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.103582 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.103483 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-kube-api-access-fq4z4" pod "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" (UID: "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6"). InnerVolumeSpecName "kube-api-access-fq4z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:30.104688 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.104312 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-envoy" pod "699f83a5-7fa4-42be-b125-c11e085ca71e" (UID: "699f83a5-7fa4-42be-b125-c11e085ca71e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.104688 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.104427 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-kube-api-access-ptvdn" pod "b9c26c24-a4ba-4ec2-9b68-e38872fa1726" (UID: "b9c26c24-a4ba-4ec2-9b68-e38872fa1726"). InnerVolumeSpecName "kube-api-access-ptvdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:30.104688 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.104542 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-kube-api-access-kd5wf" pod "699f83a5-7fa4-42be-b125-c11e085ca71e" (UID: "699f83a5-7fa4-42be-b125-c11e085ca71e"). InnerVolumeSpecName "kube-api-access-kd5wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:30.104688 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.104635 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-envoy" pod "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" (UID: "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:30.104959 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.104735 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-token" pod "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" (UID: "05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:30.104959 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.104825 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-token" pod "b9c26c24-a4ba-4ec2-9b68-e38872fa1726" (UID: "b9c26c24-a4ba-4ec2-9b68-e38872fa1726"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:30.105067 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.104957 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-token" pod "699f83a5-7fa4-42be-b125-c11e085ca71e" (UID: "699f83a5-7fa4-42be-b125-c11e085ca71e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:30.105895 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.105867 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-podinfo" pod "699f83a5-7fa4-42be-b125-c11e085ca71e" (UID: "699f83a5-7fa4-42be-b125-c11e085ca71e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.214986 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215029 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215046 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq4z4\" (UniqueName: \"kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-kube-api-access-fq4z4\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215065 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215081 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215095 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptvdn\" (UniqueName: \"kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-kube-api-access-ptvdn\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215110 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215153 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215177 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/699f83a5-7fa4-42be-b125-c11e085ca71e-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215195 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215220 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215240 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215262 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215282 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd5wf\" (UniqueName: \"kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-kube-api-access-kd5wf\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215304 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/699f83a5-7fa4-42be-b125-c11e085ca71e-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215324 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.215341 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c26c24-a4ba-4ec2-9b68-e38872fa1726-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.283221 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:30.284146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.283283 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:30.388956 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.388922 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l"] Nov 04 01:28:30.395153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.395109 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-w695l"] Nov 04 01:28:30.507315 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.507280 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef17f99d-84f8-4f1c-afbe-266df72a8a99" path="/var/lib/kubelet/pods/ef17f99d-84f8-4f1c-afbe-266df72a8a99/volumes" Nov 04 01:28:30.618857 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.618796 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" podStartSLOduration=6.618775116 podStartE2EDuration="6.618775116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:30.617166776 +0000 UTC m=+1856.233512539" watchObservedRunningTime="2026-11-04 01:28:30.618775116 +0000 UTC m=+1856.235120892" Nov 04 01:28:30.810596 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.810511 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb"] Nov 04 01:28:30.816412 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.816332 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2ee045-6d5477cfdf-dhvkb"] Nov 04 01:28:30.897836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.897758 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" Nov 04 01:28:30.898264 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.897756 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8" event={"ID":"b9c26c24-a4ba-4ec2-9b68-e38872fa1726","Type":"ContainerDied","Data":"d64660be0b29e584713463acffbd1437b9e27675cb6ee59ccd38dd491d3a8fab"} Nov 04 01:28:30.898264 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.897900 5786 scope.go:122] "RemoveContainer" containerID="595fdf925fe7979826a2ddeee1f740d49404bbe3d7e688cffb389d64a4cf6903" Nov 04 01:28:30.900841 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.900824 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" Nov 04 01:28:30.900958 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.900848 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k" event={"ID":"05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6","Type":"ContainerDied","Data":"c9560de64c7653d40c4e8d641935238dd1940cd9c1eb9bf2f011e33ceed860e6"} Nov 04 01:28:30.903971 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.903577 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" event={"ID":"699f83a5-7fa4-42be-b125-c11e085ca71e","Type":"ContainerDied","Data":"0cd1eca66d8c11cf2bdec6a5eb11af0f3ec51448f7492aa3d5c28fe31baf461e"} Nov 04 01:28:30.903971 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.903569 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2" Nov 04 01:28:30.909434 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.909394 5786 scope.go:122] "RemoveContainer" containerID="a90e461a8ce7893a989c8e45fa7ff49b6f5dd72b6787fb771d6968096bb1abe4" Nov 04 01:28:30.923107 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:30.923072 5786 scope.go:122] "RemoveContainer" containerID="fb4d659c33c19af3fdf8cdcc6030405e6700122636e173ee9ab49a72b59e6583" Nov 04 01:28:31.171159 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:31.171015 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn"] Nov 04 01:28:31.209205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:31.206193 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-ldwqn"] Nov 04 01:28:31.288148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:31.287165 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2"] Nov 04 01:28:31.291592 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:31.291563 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-krcr2"] Nov 04 01:28:31.522924 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:31.522839 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw"] Nov 04 01:28:31.526177 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:31.526151 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-pwldw"] Nov 04 01:28:31.688305 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:31.687181 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k"] Nov 04 01:28:31.692643 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:31.692611 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-dhm7k"] Nov 04 01:28:31.789387 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:31.789300 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8"] Nov 04 01:28:31.793131 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:31.793095 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-qzhk8"] Nov 04 01:28:31.897799 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:31.897769 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2"] Nov 04 01:28:31.906854 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:31.906822 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-r6rt2"] Nov 04 01:28:32.507892 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:32.507859 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" path="/var/lib/kubelet/pods/05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6/volumes" Nov 04 01:28:32.508438 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:32.508422 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061cb628-1307-41db-b238-e432a72f9150" path="/var/lib/kubelet/pods/061cb628-1307-41db-b238-e432a72f9150/volumes" Nov 04 01:28:32.508867 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:32.508855 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" path="/var/lib/kubelet/pods/156e3df3-cacc-4b41-9bf5-b2f2d481f57b/volumes" Nov 04 01:28:32.509593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:32.509578 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" path="/var/lib/kubelet/pods/4df90e93-d322-4d8e-87b3-74c38bd2e428/volumes" Nov 04 01:28:32.509995 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:32.509983 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" path="/var/lib/kubelet/pods/699f83a5-7fa4-42be-b125-c11e085ca71e/volumes" Nov 04 01:28:32.510399 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:32.510386 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c067af-133f-494e-b755-926e1047f874" path="/var/lib/kubelet/pods/72c067af-133f-494e-b755-926e1047f874/volumes" Nov 04 01:28:32.511108 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:32.511095 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" path="/var/lib/kubelet/pods/b9c26c24-a4ba-4ec2-9b68-e38872fa1726/volumes" Nov 04 01:28:34.662680 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:34.662638 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" podUID="9566986c-b6c8-4785-91c7-34f0193b9b0c" containerName="payload-processing" containerID="cri-o://6328365cb914640034601411e576bf3d86d7fe45f7173dc0788e93f59dff7698" gracePeriod=25 Nov 04 01:28:34.766812 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:34.766769 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" podUID="70495ea8-fbc0-4417-933f-14370aaacd8d" containerName="payload-pre-processing" containerID="cri-o://240226728b79b7a1e8836462deb843c811ad923fdb1f25e07685f98c44bbf8ea" gracePeriod=25 Nov 04 01:28:34.946152 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:34.945899 5786 generic.go:356] "Generic (PLEG): container finished" podID="9566986c-b6c8-4785-91c7-34f0193b9b0c" containerID="6328365cb914640034601411e576bf3d86d7fe45f7173dc0788e93f59dff7698" exitCode=0 Nov 04 01:28:34.946152 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:34.946049 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" event={"ID":"9566986c-b6c8-4785-91c7-34f0193b9b0c","Type":"ContainerDied","Data":"6328365cb914640034601411e576bf3d86d7fe45f7173dc0788e93f59dff7698"} Nov 04 01:28:34.946152 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:34.946088 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" event={"ID":"9566986c-b6c8-4785-91c7-34f0193b9b0c","Type":"ContainerDied","Data":"9310b9857644edc4933669afce1c0656e92efa6bdcbcb61a7ffe13750aa79619"} Nov 04 01:28:34.946152 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:34.946104 5786 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9310b9857644edc4933669afce1c0656e92efa6bdcbcb61a7ffe13750aa79619" Nov 04 01:28:34.949280 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:34.948943 5786 generic.go:356] "Generic (PLEG): container finished" podID="70495ea8-fbc0-4417-933f-14370aaacd8d" containerID="240226728b79b7a1e8836462deb843c811ad923fdb1f25e07685f98c44bbf8ea" exitCode=0 Nov 04 01:28:34.949280 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:34.949081 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" event={"ID":"70495ea8-fbc0-4417-933f-14370aaacd8d","Type":"ContainerDied","Data":"240226728b79b7a1e8836462deb843c811ad923fdb1f25e07685f98c44bbf8ea"} Nov 04 01:28:34.965288 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:34.965257 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" Nov 04 01:28:35.063922 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.063896 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" Nov 04 01:28:35.136374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.136338 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9566986c-b6c8-4785-91c7-34f0193b9b0c-kube-api-access-bt2cj\" (UniqueName: \"kubernetes.io/projected/9566986c-b6c8-4785-91c7-34f0193b9b0c-kube-api-access-bt2cj\") pod \"9566986c-b6c8-4785-91c7-34f0193b9b0c\" (UID: \"9566986c-b6c8-4785-91c7-34f0193b9b0c\") " Nov 04 01:28:35.136597 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.136576 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9566986c-b6c8-4785-91c7-34f0193b9b0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9566986c-b6c8-4785-91c7-34f0193b9b0c-plugins-config-volume\") pod \"9566986c-b6c8-4785-91c7-34f0193b9b0c\" (UID: \"9566986c-b6c8-4785-91c7-34f0193b9b0c\") " Nov 04 01:28:35.136928 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.136905 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9566986c-b6c8-4785-91c7-34f0193b9b0c-plugins-config-volume" pod "9566986c-b6c8-4785-91c7-34f0193b9b0c" (UID: "9566986c-b6c8-4785-91c7-34f0193b9b0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:35.137086 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.137070 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9566986c-b6c8-4785-91c7-34f0193b9b0c-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:35.138387 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.138369 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9566986c-b6c8-4785-91c7-34f0193b9b0c-kube-api-access-bt2cj" pod "9566986c-b6c8-4785-91c7-34f0193b9b0c" (UID: "9566986c-b6c8-4785-91c7-34f0193b9b0c"). InnerVolumeSpecName "kube-api-access-bt2cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:35.238265 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.238153 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70495ea8-fbc0-4417-933f-14370aaacd8d-kube-api-access-gkdqm\" (UniqueName: \"kubernetes.io/projected/70495ea8-fbc0-4417-933f-14370aaacd8d-kube-api-access-gkdqm\") pod \"70495ea8-fbc0-4417-933f-14370aaacd8d\" (UID: \"70495ea8-fbc0-4417-933f-14370aaacd8d\") " Nov 04 01:28:35.238449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.238429 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70495ea8-fbc0-4417-933f-14370aaacd8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70495ea8-fbc0-4417-933f-14370aaacd8d-plugins-config-volume\") pod \"70495ea8-fbc0-4417-933f-14370aaacd8d\" (UID: \"70495ea8-fbc0-4417-933f-14370aaacd8d\") " Nov 04 01:28:35.238807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.238779 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70495ea8-fbc0-4417-933f-14370aaacd8d-plugins-config-volume" pod "70495ea8-fbc0-4417-933f-14370aaacd8d" (UID: "70495ea8-fbc0-4417-933f-14370aaacd8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:35.238983 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.238881 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70495ea8-fbc0-4417-933f-14370aaacd8d-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:35.238983 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.238895 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt2cj\" (UniqueName: \"kubernetes.io/projected/9566986c-b6c8-4785-91c7-34f0193b9b0c-kube-api-access-bt2cj\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:35.240377 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.240350 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70495ea8-fbc0-4417-933f-14370aaacd8d-kube-api-access-gkdqm" pod "70495ea8-fbc0-4417-933f-14370aaacd8d" (UID: "70495ea8-fbc0-4417-933f-14370aaacd8d"). InnerVolumeSpecName "kube-api-access-gkdqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:35.340891 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.340862 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkdqm\" (UniqueName: \"kubernetes.io/projected/70495ea8-fbc0-4417-933f-14370aaacd8d-kube-api-access-gkdqm\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:35.957154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.957042 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" Nov 04 01:28:35.957154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.957060 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl" event={"ID":"70495ea8-fbc0-4417-933f-14370aaacd8d","Type":"ContainerDied","Data":"c77eaadf4a17670db136abee2232bccd2054e68ef83f540fa0da9fa18cb5b46c"} Nov 04 01:28:35.957637 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.957168 5786 scope.go:122] "RemoveContainer" containerID="240226728b79b7a1e8836462deb843c811ad923fdb1f25e07685f98c44bbf8ea" Nov 04 01:28:35.957637 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:35.957324 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq" Nov 04 01:28:36.131564 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:36.131529 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl"] Nov 04 01:28:36.136024 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:36.136001 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8aa9f3a2-75cccdb575-nklfl"] Nov 04 01:28:36.292419 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:36.292310 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq"] Nov 04 01:28:36.295213 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:36.295186 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8aa9f3a2-686b95cb6b-fcvkq"] Nov 04 01:28:36.506637 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:36.506605 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70495ea8-fbc0-4417-933f-14370aaacd8d" path="/var/lib/kubelet/pods/70495ea8-fbc0-4417-933f-14370aaacd8d/volumes" Nov 04 01:28:36.507074 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:36.507059 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9566986c-b6c8-4785-91c7-34f0193b9b0c" path="/var/lib/kubelet/pods/9566986c-b6c8-4785-91c7-34f0193b9b0c/volumes" Nov 04 01:28:37.342541 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.342508 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7"] Nov 04 01:28:37.343183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343166 5786 cpu_manager.go:414] "containerMap: removing container" podUID="72c067af-133f-494e-b755-926e1047f874" containerName="maas-api" Nov 04 01:28:37.343183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343184 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c067af-133f-494e-b755-926e1047f874" containerName="maas-api" Nov 04 01:28:37.343272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343198 5786 cpu_manager.go:414] "containerMap: removing container" podUID="061cb628-1307-41db-b238-e432a72f9150" containerName="istio-proxy" Nov 04 01:28:37.343272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343203 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061cb628-1307-41db-b238-e432a72f9150" containerName="istio-proxy" Nov 04 01:28:37.343272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343215 5786 cpu_manager.go:414] "containerMap: removing container" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerName="istio-proxy" Nov 04 01:28:37.343272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343222 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerName="istio-proxy" Nov 04 01:28:37.343272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343241 5786 cpu_manager.go:414] "containerMap: removing container" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerName="istio-proxy" Nov 04 01:28:37.343272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343247 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerName="istio-proxy" Nov 04 01:28:37.343272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343257 5786 cpu_manager.go:414] "containerMap: removing container" podUID="70495ea8-fbc0-4417-933f-14370aaacd8d" containerName="payload-pre-processing" Nov 04 01:28:37.343272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343261 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70495ea8-fbc0-4417-933f-14370aaacd8d" containerName="payload-pre-processing" Nov 04 01:28:37.343272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343274 5786 cpu_manager.go:414] "containerMap: removing container" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343279 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343291 5786 cpu_manager.go:414] "containerMap: removing container" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343296 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343308 5786 cpu_manager.go:414] "containerMap: removing container" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343312 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343319 5786 cpu_manager.go:414] "containerMap: removing container" podUID="9566986c-b6c8-4785-91c7-34f0193b9b0c" containerName="payload-processing" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343323 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9566986c-b6c8-4785-91c7-34f0193b9b0c" containerName="payload-processing" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343331 5786 cpu_manager.go:414] "containerMap: removing container" podUID="ef17f99d-84f8-4f1c-afbe-266df72a8a99" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343336 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef17f99d-84f8-4f1c-afbe-266df72a8a99" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343426 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="156e3df3-cacc-4b41-9bf5-b2f2d481f57b" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343436 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="9566986c-b6c8-4785-91c7-34f0193b9b0c" containerName="payload-processing" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343443 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef17f99d-84f8-4f1c-afbe-266df72a8a99" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343449 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c067af-133f-494e-b755-926e1047f874" containerName="maas-api" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343456 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="061cb628-1307-41db-b238-e432a72f9150" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343463 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dfa65e-31a1-4b7f-9bdf-eb22512fa3c6" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343472 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c26c24-a4ba-4ec2-9b68-e38872fa1726" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343479 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="70495ea8-fbc0-4417-933f-14370aaacd8d" containerName="payload-pre-processing" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343485 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="699f83a5-7fa4-42be-b125-c11e085ca71e" containerName="istio-proxy" Nov 04 01:28:37.343514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.343492 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df90e93-d322-4d8e-87b3-74c38bd2e428" containerName="istio-proxy" Nov 04 01:28:37.348532 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.348504 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" Nov 04 01:28:37.403049 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.403016 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7"] Nov 04 01:28:37.403536 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.403516 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-42958c\"" Nov 04 01:28:37.403747 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.403724 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-42958c-dockercfg-krn9x\"" Nov 04 01:28:37.414412 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.414384 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmng9\" (UniqueName: \"kubernetes.io/projected/97d6013a-d49b-4639-88dd-f5e458f50d28-kube-api-access-lmng9\") pod \"payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7\" (UID: \"97d6013a-d49b-4639-88dd-f5e458f50d28\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" Nov 04 01:28:37.414583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.414538 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d6013a-d49b-4639-88dd-f5e458f50d28-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7\" (UID: \"97d6013a-d49b-4639-88dd-f5e458f50d28\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" Nov 04 01:28:37.546143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.543200 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d6013a-d49b-4639-88dd-f5e458f50d28-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7\" (UID: \"97d6013a-d49b-4639-88dd-f5e458f50d28\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" Nov 04 01:28:37.546143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.543570 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmng9\" (UniqueName: \"kubernetes.io/projected/97d6013a-d49b-4639-88dd-f5e458f50d28-kube-api-access-lmng9\") pod \"payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7\" (UID: \"97d6013a-d49b-4639-88dd-f5e458f50d28\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" Nov 04 01:28:37.546143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.544897 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d6013a-d49b-4639-88dd-f5e458f50d28-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7\" (UID: \"97d6013a-d49b-4639-88dd-f5e458f50d28\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" Nov 04 01:28:37.661461 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.659752 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmng9\" (UniqueName: \"kubernetes.io/projected/97d6013a-d49b-4639-88dd-f5e458f50d28-kube-api-access-lmng9\") pod \"payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7\" (UID: \"97d6013a-d49b-4639-88dd-f5e458f50d28\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" Nov 04 01:28:37.677179 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.665497 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" Nov 04 01:28:37.677179 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.675865 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm"] Nov 04 01:28:37.688398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.683432 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" Nov 04 01:28:37.739951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.739895 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm"] Nov 04 01:28:37.851990 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.851570 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhh5h\" (UniqueName: \"kubernetes.io/projected/d198877f-a0af-4c3a-8902-160811220bd8-kube-api-access-vhh5h\") pod \"payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm\" (UID: \"d198877f-a0af-4c3a-8902-160811220bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" Nov 04 01:28:37.851990 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.851779 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d198877f-a0af-4c3a-8902-160811220bd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm\" (UID: \"d198877f-a0af-4c3a-8902-160811220bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" Nov 04 01:28:37.952955 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.952863 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhh5h\" (UniqueName: \"kubernetes.io/projected/d198877f-a0af-4c3a-8902-160811220bd8-kube-api-access-vhh5h\") pod \"payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm\" (UID: \"d198877f-a0af-4c3a-8902-160811220bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" Nov 04 01:28:37.953131 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.952983 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d198877f-a0af-4c3a-8902-160811220bd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm\" (UID: \"d198877f-a0af-4c3a-8902-160811220bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" Nov 04 01:28:37.953678 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:37.953657 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d198877f-a0af-4c3a-8902-160811220bd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm\" (UID: \"d198877f-a0af-4c3a-8902-160811220bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" Nov 04 01:28:38.009719 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:38.009691 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhh5h\" (UniqueName: \"kubernetes.io/projected/d198877f-a0af-4c3a-8902-160811220bd8-kube-api-access-vhh5h\") pod \"payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm\" (UID: \"d198877f-a0af-4c3a-8902-160811220bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" Nov 04 01:28:38.028671 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:38.028643 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" Nov 04 01:28:38.132642 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:38.132608 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7"] Nov 04 01:28:38.419928 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:38.419767 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm"] Nov 04 01:28:38.423213 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:28:38.423182 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd198877f_a0af_4c3a_8902_160811220bd8.slice/crio-17e0ab8076e94b49c25afccb0a63ce03ad22a226d95e727e5a598a4f5b89238b WatchSource:0}: Error finding container 17e0ab8076e94b49c25afccb0a63ce03ad22a226d95e727e5a598a4f5b89238b: Status 404 returned error can't find the container with id 17e0ab8076e94b49c25afccb0a63ce03ad22a226d95e727e5a598a4f5b89238b Nov 04 01:28:38.986244 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:38.986209 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp"] Nov 04 01:28:38.986486 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:38.986452 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" containerID="cri-o://970d375edc0af449a68cd1bf961f32c398c313fa3f3410aeac3b30385d6bc2b8" gracePeriod=30 Nov 04 01:28:38.989922 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:38.989897 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4"] Nov 04 01:28:38.990145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:38.990098 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" containerID="cri-o://01865512c7354a34e7883e11e1c85f0c16a34c3eab7b1aabcfca81d6d458eed6" gracePeriod=30 Nov 04 01:28:39.005255 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:39.005224 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s"] Nov 04 01:28:39.005427 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:39.005406 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" containerID="cri-o://041de15dd17b778b4cd24cfb93805c01067fa510237f85f4fb028017d0adcd69" gracePeriod=30 Nov 04 01:28:39.031323 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:39.031292 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g"] Nov 04 01:28:39.031508 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:39.031485 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" podUID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" containerName="istio-proxy" containerID="cri-o://bfc49586d872509d524d1a106fb72b90a9dcb19c6469dcea49937c43d2a13e19" gracePeriod=30 Nov 04 01:28:39.324584 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:39.324498 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" event={"ID":"97d6013a-d49b-4639-88dd-f5e458f50d28","Type":"ContainerStarted","Data":"05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b"} Nov 04 01:28:39.324735 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:39.324612 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" Nov 04 01:28:39.324735 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:39.324632 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" event={"ID":"97d6013a-d49b-4639-88dd-f5e458f50d28","Type":"ContainerStarted","Data":"598556e0bfe41f3262c973e29fdc69c0c0ba1818fdc9b8ea102b40d0553b8e37"} Nov 04 01:28:39.326075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:39.326052 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" event={"ID":"d198877f-a0af-4c3a-8902-160811220bd8","Type":"ContainerStarted","Data":"c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f"} Nov 04 01:28:39.326205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:39.326093 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" event={"ID":"d198877f-a0af-4c3a-8902-160811220bd8","Type":"ContainerStarted","Data":"17e0ab8076e94b49c25afccb0a63ce03ad22a226d95e727e5a598a4f5b89238b"} Nov 04 01:28:39.326311 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:39.326298 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" Nov 04 01:28:39.659755 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:39.659698 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" podStartSLOduration=2.659684577 podStartE2EDuration="2.659684577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:39.659114119 +0000 UTC m=+1865.275459882" watchObservedRunningTime="2026-11-04 01:28:39.659684577 +0000 UTC m=+1865.276030337" Nov 04 01:28:39.848141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:39.838699 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" podStartSLOduration=2.838673487 podStartE2EDuration="2.838673487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:39.794538173 +0000 UTC m=+1865.410883936" watchObservedRunningTime="2026-11-04 01:28:39.838673487 +0000 UTC m=+1865.455019251" Nov 04 01:28:40.909274 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:40.909245 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" Nov 04 01:28:40.909667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:40.909313 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" Nov 04 01:28:41.423636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:41.423601 5786 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:41.423809 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:41.423658 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:44.405686 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.405459 5786 generic.go:356] "Generic (PLEG): container finished" podID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" containerID="bfc49586d872509d524d1a106fb72b90a9dcb19c6469dcea49937c43d2a13e19" exitCode=0 Nov 04 01:28:44.405686 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.405596 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" event={"ID":"95ca1b1e-0ca9-485f-8f2b-1d013b618b66","Type":"ContainerDied","Data":"bfc49586d872509d524d1a106fb72b90a9dcb19c6469dcea49937c43d2a13e19"} Nov 04 01:28:44.409060 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.408768 5786 generic.go:356] "Generic (PLEG): container finished" podID="51822598-8499-472e-b731-4573487376e9" containerID="01865512c7354a34e7883e11e1c85f0c16a34c3eab7b1aabcfca81d6d458eed6" exitCode=0 Nov 04 01:28:44.409060 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.408872 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" event={"ID":"51822598-8499-472e-b731-4573487376e9","Type":"ContainerDied","Data":"01865512c7354a34e7883e11e1c85f0c16a34c3eab7b1aabcfca81d6d458eed6"} Nov 04 01:28:44.409060 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.408930 5786 scope.go:122] "RemoveContainer" containerID="1489b1ed2688aa52fdd17db49523ad7248b665f48f81155aaaad20719b277eb9" Nov 04 01:28:44.412657 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.412443 5786 generic.go:356] "Generic (PLEG): container finished" podID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerID="041de15dd17b778b4cd24cfb93805c01067fa510237f85f4fb028017d0adcd69" exitCode=0 Nov 04 01:28:44.412657 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.412585 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" event={"ID":"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6","Type":"ContainerDied","Data":"041de15dd17b778b4cd24cfb93805c01067fa510237f85f4fb028017d0adcd69"} Nov 04 01:28:44.415981 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.415823 5786 generic.go:356] "Generic (PLEG): container finished" podID="899e8d41-45fd-4178-95d8-271b97510499" containerID="970d375edc0af449a68cd1bf961f32c398c313fa3f3410aeac3b30385d6bc2b8" exitCode=0 Nov 04 01:28:44.415981 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.415912 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" event={"ID":"899e8d41-45fd-4178-95d8-271b97510499","Type":"ContainerDied","Data":"970d375edc0af449a68cd1bf961f32c398c313fa3f3410aeac3b30385d6bc2b8"} Nov 04 01:28:44.488296 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.488265 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:28:44.491511 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.491476 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:28:44.556335 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.556314 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:28:44.561249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.561226 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:28:44.562264 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562239 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-socket\") pod \"51822598-8499-472e-b731-4573487376e9\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " Nov 04 01:28:44.562382 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562278 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istiod-ca-cert\") pod \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " Nov 04 01:28:44.562382 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562332 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-data\") pod \"51822598-8499-472e-b731-4573487376e9\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " Nov 04 01:28:44.562382 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562374 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/899e8d41-45fd-4178-95d8-271b97510499-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899e8d41-45fd-4178-95d8-271b97510499-istiod-ca-cert\") pod \"899e8d41-45fd-4178-95d8-271b97510499\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " Nov 04 01:28:44.562539 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562417 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-certs\") pod \"899e8d41-45fd-4178-95d8-271b97510499\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " Nov 04 01:28:44.562539 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562449 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-token\" (UniqueName: \"kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-token\") pod \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " Nov 04 01:28:44.562539 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562488 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-kube-api-access-dwcxg\" (UniqueName: \"kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-kube-api-access-dwcxg\") pod \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " Nov 04 01:28:44.562539 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562494 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-socket" pod "51822598-8499-472e-b731-4573487376e9" (UID: "51822598-8499-472e-b731-4573487376e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.562539 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562519 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-kube-api-access-kzj9d\" (UniqueName: \"kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-kube-api-access-kzj9d\") pod \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " Nov 04 01:28:44.562788 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562557 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-credential-socket\") pod \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " Nov 04 01:28:44.562788 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562594 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-socket\") pod \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " Nov 04 01:28:44.562788 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562648 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-token\" (UniqueName: \"kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-token\") pod \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " Nov 04 01:28:44.562788 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562644 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-data" pod "51822598-8499-472e-b731-4573487376e9" (UID: "51822598-8499-472e-b731-4573487376e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.562788 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562683 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istiod-ca-cert\") pod \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " Nov 04 01:28:44.562788 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562711 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-envoy\") pod \"51822598-8499-472e-b731-4573487376e9\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " Nov 04 01:28:44.562788 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562741 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-certs\") pod \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " Nov 04 01:28:44.562788 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562772 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-socket\") pod \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " Nov 04 01:28:44.563183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562804 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-kube-api-access-4266x\" (UniqueName: \"kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-kube-api-access-4266x\") pod \"899e8d41-45fd-4178-95d8-271b97510499\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " Nov 04 01:28:44.563183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562839 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-certs\") pod \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " Nov 04 01:28:44.563183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562867 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-certs\") pod \"51822598-8499-472e-b731-4573487376e9\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " Nov 04 01:28:44.563183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562870 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-credential-socket" pod "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" (UID: "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.563183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562794 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899e8d41-45fd-4178-95d8-271b97510499-istiod-ca-cert" pod "899e8d41-45fd-4178-95d8-271b97510499" (UID: "899e8d41-45fd-4178-95d8-271b97510499"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:44.563183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562895 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-data\") pod \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " Nov 04 01:28:44.563183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562928 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51822598-8499-472e-b731-4573487376e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51822598-8499-472e-b731-4573487376e9-istio-podinfo\") pod \"51822598-8499-472e-b731-4573487376e9\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " Nov 04 01:28:44.563183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562960 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-istio-token\" (UniqueName: \"kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-istio-token\") pod \"51822598-8499-472e-b731-4573487376e9\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " Nov 04 01:28:44.563183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.562996 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-socket\") pod \"899e8d41-45fd-4178-95d8-271b97510499\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " Nov 04 01:28:44.563183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563060 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-credential-socket\") pod \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " Nov 04 01:28:44.563183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563099 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-podinfo\") pod \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563202 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-envoy\") pod \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563264 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-credential-socket\") pod \"51822598-8499-472e-b731-4573487376e9\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563299 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-istio-token\" (UniqueName: \"kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-istio-token\") pod \"899e8d41-45fd-4178-95d8-271b97510499\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563341 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-podinfo\") pod \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563375 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-credential-socket\") pod \"899e8d41-45fd-4178-95d8-271b97510499\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563386 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istiod-ca-cert" pod "95ca1b1e-0ca9-485f-8f2b-1d013b618b66" (UID: "95ca1b1e-0ca9-485f-8f2b-1d013b618b66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563470 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-kube-api-access-z9kvm\" (UniqueName: \"kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-kube-api-access-z9kvm\") pod \"51822598-8499-472e-b731-4573487376e9\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563509 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-data\") pod \"899e8d41-45fd-4178-95d8-271b97510499\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563559 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-envoy\") pod \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\" (UID: \"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6\") " Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563641 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/899e8d41-45fd-4178-95d8-271b97510499-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899e8d41-45fd-4178-95d8-271b97510499-istio-podinfo\") pod \"899e8d41-45fd-4178-95d8-271b97510499\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563653 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-data" pod "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" (UID: "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563665 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-certs" pod "899e8d41-45fd-4178-95d8-271b97510499" (UID: "899e8d41-45fd-4178-95d8-271b97510499"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.563704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563671 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-data\") pod \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\" (UID: \"95ca1b1e-0ca9-485f-8f2b-1d013b618b66\") " Nov 04 01:28:44.564423 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563855 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-socket" pod "95ca1b1e-0ca9-485f-8f2b-1d013b618b66" (UID: "95ca1b1e-0ca9-485f-8f2b-1d013b618b66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.564423 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563876 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-credential-socket" pod "51822598-8499-472e-b731-4573487376e9" (UID: "51822598-8499-472e-b731-4573487376e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.564423 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563911 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-socket" pod "899e8d41-45fd-4178-95d8-271b97510499" (UID: "899e8d41-45fd-4178-95d8-271b97510499"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.564423 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.563919 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-certs" pod "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" (UID: "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.564423 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.564048 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-socket" pod "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" (UID: "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.564423 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.564061 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-credential-socket" pod "95ca1b1e-0ca9-485f-8f2b-1d013b618b66" (UID: "95ca1b1e-0ca9-485f-8f2b-1d013b618b66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.564423 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.564245 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-envoy\") pod \"899e8d41-45fd-4178-95d8-271b97510499\" (UID: \"899e8d41-45fd-4178-95d8-271b97510499\") " Nov 04 01:28:44.564423 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.564292 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51822598-8499-472e-b731-4573487376e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51822598-8499-472e-b731-4573487376e9-istiod-ca-cert\") pod \"51822598-8499-472e-b731-4573487376e9\" (UID: \"51822598-8499-472e-b731-4573487376e9\") " Nov 04 01:28:44.564851 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.564825 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-credential-socket" pod "899e8d41-45fd-4178-95d8-271b97510499" (UID: "899e8d41-45fd-4178-95d8-271b97510499"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.565098 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565072 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899e8d41-45fd-4178-95d8-271b97510499-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565104 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565140 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565156 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565171 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565185 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565198 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565212 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565226 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565240 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565256 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565271 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565285 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565298 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.565514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565234 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-token" pod "95ca1b1e-0ca9-485f-8f2b-1d013b618b66" (UID: "95ca1b1e-0ca9-485f-8f2b-1d013b618b66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.565514 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.565450 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istiod-ca-cert" pod "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" (UID: "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:44.566156 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.566109 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-kube-api-access-dwcxg" pod "95ca1b1e-0ca9-485f-8f2b-1d013b618b66" (UID: "95ca1b1e-0ca9-485f-8f2b-1d013b618b66"). InnerVolumeSpecName "kube-api-access-dwcxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.568578 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.566969 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-data" pod "95ca1b1e-0ca9-485f-8f2b-1d013b618b66" (UID: "95ca1b1e-0ca9-485f-8f2b-1d013b618b66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.568578 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.567954 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-envoy" pod "51822598-8499-472e-b731-4573487376e9" (UID: "51822598-8499-472e-b731-4573487376e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.568578 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.568288 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51822598-8499-472e-b731-4573487376e9-istiod-ca-cert" pod "51822598-8499-472e-b731-4573487376e9" (UID: "51822598-8499-472e-b731-4573487376e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:44.568578 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.568365 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-certs" pod "51822598-8499-472e-b731-4573487376e9" (UID: "51822598-8499-472e-b731-4573487376e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.568578 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.568437 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-certs" pod "95ca1b1e-0ca9-485f-8f2b-1d013b618b66" (UID: "95ca1b1e-0ca9-485f-8f2b-1d013b618b66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.568578 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.568554 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-data" pod "899e8d41-45fd-4178-95d8-271b97510499" (UID: "899e8d41-45fd-4178-95d8-271b97510499"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.569286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.569257 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-podinfo" pod "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" (UID: "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:44.569520 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.569482 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-kube-api-access-kzj9d" pod "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" (UID: "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6"). InnerVolumeSpecName "kube-api-access-kzj9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.569787 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.569760 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-envoy" pod "95ca1b1e-0ca9-485f-8f2b-1d013b618b66" (UID: "95ca1b1e-0ca9-485f-8f2b-1d013b618b66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.569878 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.569795 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-kube-api-access-4266x" pod "899e8d41-45fd-4178-95d8-271b97510499" (UID: "899e8d41-45fd-4178-95d8-271b97510499"). InnerVolumeSpecName "kube-api-access-4266x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.569878 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.569829 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-istio-token" pod "899e8d41-45fd-4178-95d8-271b97510499" (UID: "899e8d41-45fd-4178-95d8-271b97510499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.569878 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.569829 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/899e8d41-45fd-4178-95d8-271b97510499-istio-podinfo" pod "899e8d41-45fd-4178-95d8-271b97510499" (UID: "899e8d41-45fd-4178-95d8-271b97510499"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:44.569878 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.569866 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-envoy" pod "899e8d41-45fd-4178-95d8-271b97510499" (UID: "899e8d41-45fd-4178-95d8-271b97510499"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.569878 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.569872 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51822598-8499-472e-b731-4573487376e9-istio-podinfo" pod "51822598-8499-472e-b731-4573487376e9" (UID: "51822598-8499-472e-b731-4573487376e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:44.570051 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.569881 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-istio-token" pod "51822598-8499-472e-b731-4573487376e9" (UID: "51822598-8499-472e-b731-4573487376e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.570051 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.569891 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-token" pod "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" (UID: "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.570051 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.569981 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-podinfo" pod "95ca1b1e-0ca9-485f-8f2b-1d013b618b66" (UID: "95ca1b1e-0ca9-485f-8f2b-1d013b618b66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:44.570988 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.570969 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-kube-api-access-z9kvm" pod "51822598-8499-472e-b731-4573487376e9" (UID: "51822598-8499-472e-b731-4573487376e9"). InnerVolumeSpecName "kube-api-access-z9kvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.571633 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.571611 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-envoy" pod "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" (UID: "5a9235dc-6d4e-4141-b20d-3b17bbeb28b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.667611 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667464 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.667611 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667499 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwcxg\" (UniqueName: \"kubernetes.io/projected/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-kube-api-access-dwcxg\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.667611 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667517 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzj9d\" (UniqueName: \"kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-kube-api-access-kzj9d\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.667611 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667531 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.667611 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667547 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.667611 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667561 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.667611 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667575 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4266x\" (UniqueName: \"kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-kube-api-access-4266x\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.667611 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667589 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.667611 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667603 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51822598-8499-472e-b731-4573487376e9-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.667611 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667615 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51822598-8499-472e-b731-4573487376e9-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.668249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667628 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.668249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667640 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.668249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667653 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.668249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667665 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899e8d41-45fd-4178-95d8-271b97510499-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.668249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667679 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.668249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667695 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9kvm\" (UniqueName: \"kubernetes.io/projected/51822598-8499-472e-b731-4573487376e9-kube-api-access-z9kvm\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.668249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667709 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.668249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667725 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.668249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667738 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899e8d41-45fd-4178-95d8-271b97510499-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.668249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667751 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ca1b1e-0ca9-485f-8f2b-1d013b618b66-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.668249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667765 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899e8d41-45fd-4178-95d8-271b97510499-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.668249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:44.667779 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51822598-8499-472e-b731-4573487376e9-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:45.090196 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:45.090157 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:28:45.424482 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.424452 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" Nov 04 01:28:45.424951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.424494 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g" event={"ID":"95ca1b1e-0ca9-485f-8f2b-1d013b618b66","Type":"ContainerDied","Data":"bdfbd0e79eccea9c940aeec3278eac4e2be7ebbab2e91528583ccfb95a89aacc"} Nov 04 01:28:45.424951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.424557 5786 scope.go:122] "RemoveContainer" containerID="bfc49586d872509d524d1a106fb72b90a9dcb19c6469dcea49937c43d2a13e19" Nov 04 01:28:45.426355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.426325 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" event={"ID":"51822598-8499-472e-b731-4573487376e9","Type":"ContainerDied","Data":"83c9b7330fffd2b8b22c779ff5c945c3cff769358112104717cb09b8065c70da"} Nov 04 01:28:45.426465 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.426370 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4" Nov 04 01:28:45.428103 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.428017 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" Nov 04 01:28:45.428103 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.428044 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s" event={"ID":"5a9235dc-6d4e-4141-b20d-3b17bbeb28b6","Type":"ContainerDied","Data":"8ab4795ea838c74a8d1ed9a12b25c73a9188a745d2c97a5c48e9c9d40c291833"} Nov 04 01:28:45.429855 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.429834 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" event={"ID":"899e8d41-45fd-4178-95d8-271b97510499","Type":"ContainerDied","Data":"9528e48ee137f03e9acb1109cfca2423c9c97ab98c6883ae1166a5066997b964"} Nov 04 01:28:45.429952 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.429926 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp" Nov 04 01:28:45.442239 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.441848 5786 scope.go:122] "RemoveContainer" containerID="01865512c7354a34e7883e11e1c85f0c16a34c3eab7b1aabcfca81d6d458eed6" Nov 04 01:28:45.455174 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.455152 5786 scope.go:122] "RemoveContainer" containerID="041de15dd17b778b4cd24cfb93805c01067fa510237f85f4fb028017d0adcd69" Nov 04 01:28:45.465198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.465175 5786 scope.go:122] "RemoveContainer" containerID="970d375edc0af449a68cd1bf961f32c398c313fa3f3410aeac3b30385d6bc2b8" Nov 04 01:28:45.586677 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.586637 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4"] Nov 04 01:28:45.634032 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.633998 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-h5xk4"] Nov 04 01:28:45.701861 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.701676 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s"] Nov 04 01:28:45.705970 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.705942 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-f2h5s"] Nov 04 01:28:45.815243 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.815204 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g"] Nov 04 01:28:45.820133 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.820094 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-slv2g"] Nov 04 01:28:45.925436 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.925400 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp"] Nov 04 01:28:45.932453 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:45.932427 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-tw5xp"] Nov 04 01:28:46.506223 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:46.506193 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51822598-8499-472e-b731-4573487376e9" path="/var/lib/kubelet/pods/51822598-8499-472e-b731-4573487376e9/volumes" Nov 04 01:28:46.506686 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:46.506671 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" path="/var/lib/kubelet/pods/5a9235dc-6d4e-4141-b20d-3b17bbeb28b6/volumes" Nov 04 01:28:46.507132 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:46.507106 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="899e8d41-45fd-4178-95d8-271b97510499" path="/var/lib/kubelet/pods/899e8d41-45fd-4178-95d8-271b97510499/volumes" Nov 04 01:28:46.507868 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:46.507853 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" path="/var/lib/kubelet/pods/95ca1b1e-0ca9-485f-8f2b-1d013b618b66/volumes" Nov 04 01:28:50.335934 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.335901 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" Nov 04 01:28:50.336418 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.335967 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" Nov 04 01:28:50.464480 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.464444 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq"] Nov 04 01:28:50.464718 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.464678 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" podUID="b4447b9a-810f-4410-909c-9d51764eff35" containerName="maas-api" containerID="cri-o://bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5" gracePeriod=30 Nov 04 01:28:50.745694 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.745667 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:28:50.855463 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.855428 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-metrics-certs\") pod \"b4447b9a-810f-4410-909c-9d51764eff35\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " Nov 04 01:28:50.855662 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.855647 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4447b9a-810f-4410-909c-9d51764eff35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4447b9a-810f-4410-909c-9d51764eff35-openshift-service-ca\") pod \"b4447b9a-810f-4410-909c-9d51764eff35\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " Nov 04 01:28:50.855775 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.855758 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4447b9a-810f-4410-909c-9d51764eff35-kube-api-access-gvfpq\" (UniqueName: \"kubernetes.io/projected/b4447b9a-810f-4410-909c-9d51764eff35-kube-api-access-gvfpq\") pod \"b4447b9a-810f-4410-909c-9d51764eff35\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " Nov 04 01:28:50.855871 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.855855 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-maas-api-tls\") pod \"b4447b9a-810f-4410-909c-9d51764eff35\" (UID: \"b4447b9a-810f-4410-909c-9d51764eff35\") " Nov 04 01:28:50.856275 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.856248 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4447b9a-810f-4410-909c-9d51764eff35-openshift-service-ca" pod "b4447b9a-810f-4410-909c-9d51764eff35" (UID: "b4447b9a-810f-4410-909c-9d51764eff35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:50.859273 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.859248 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4447b9a-810f-4410-909c-9d51764eff35-kube-api-access-gvfpq" pod "b4447b9a-810f-4410-909c-9d51764eff35" (UID: "b4447b9a-810f-4410-909c-9d51764eff35"). InnerVolumeSpecName "kube-api-access-gvfpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:50.860553 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.860532 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-metrics-certs" pod "b4447b9a-810f-4410-909c-9d51764eff35" (UID: "b4447b9a-810f-4410-909c-9d51764eff35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:50.860553 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.860546 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-maas-api-tls" pod "b4447b9a-810f-4410-909c-9d51764eff35" (UID: "b4447b9a-810f-4410-909c-9d51764eff35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:50.957618 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.957533 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4447b9a-810f-4410-909c-9d51764eff35-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:50.957618 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.957567 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvfpq\" (UniqueName: \"kubernetes.io/projected/b4447b9a-810f-4410-909c-9d51764eff35-kube-api-access-gvfpq\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:50.957618 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.957579 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:50.957618 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:50.957589 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4447b9a-810f-4410-909c-9d51764eff35-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:51.525212 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:51.514188 5786 generic.go:356] "Generic (PLEG): container finished" podID="b4447b9a-810f-4410-909c-9d51764eff35" containerID="bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5" exitCode=0 Nov 04 01:28:51.525212 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:51.514374 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" event={"ID":"b4447b9a-810f-4410-909c-9d51764eff35","Type":"ContainerDied","Data":"bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5"} Nov 04 01:28:51.525212 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:51.514425 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" event={"ID":"b4447b9a-810f-4410-909c-9d51764eff35","Type":"ContainerDied","Data":"ab9f842db2ba07bf0a42f6c8b2fb97b9c6427b9e9b9ac824ff08a7b140411f05"} Nov 04 01:28:51.525212 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:51.514454 5786 scope.go:122] "RemoveContainer" containerID="bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5" Nov 04 01:28:51.525212 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:51.514610 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq" Nov 04 01:28:51.617328 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:51.612686 5786 scope.go:122] "RemoveContainer" containerID="bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5" Nov 04 01:28:51.617328 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:51.613282 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5\": container with ID starting with bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5 not found: ID does not exist" containerID="bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5" Nov 04 01:28:51.617328 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:51.613320 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5"} err="failed to get container status \"bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5\": rpc error: code = NotFound desc = could not find container \"bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5\": container with ID starting with bcafb001e2f461f5fc65ad273774ed0edbe31157d3d5b4cb1b04195422545de5 not found: ID does not exist" Nov 04 01:28:51.736983 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:51.736949 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq"] Nov 04 01:28:51.790238 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:51.790148 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8aa9f3a2-74bcd7f9dc-jvjzq"] Nov 04 01:28:52.120367 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.120332 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k"] Nov 04 01:28:52.121237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121211 5786 cpu_manager.go:414] "containerMap: removing container" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" Nov 04 01:28:52.121237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121234 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" Nov 04 01:28:52.121469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121250 5786 cpu_manager.go:414] "containerMap: removing container" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" Nov 04 01:28:52.121469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121258 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" Nov 04 01:28:52.121469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121283 5786 cpu_manager.go:414] "containerMap: removing container" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" Nov 04 01:28:52.121469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121290 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" Nov 04 01:28:52.121469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121304 5786 cpu_manager.go:414] "containerMap: removing container" podUID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" containerName="istio-proxy" Nov 04 01:28:52.121469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121309 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" containerName="istio-proxy" Nov 04 01:28:52.121469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121322 5786 cpu_manager.go:414] "containerMap: removing container" podUID="b4447b9a-810f-4410-909c-9d51764eff35" containerName="maas-api" Nov 04 01:28:52.121469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121326 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4447b9a-810f-4410-909c-9d51764eff35" containerName="maas-api" Nov 04 01:28:52.121469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121466 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" Nov 04 01:28:52.121469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121475 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9235dc-6d4e-4141-b20d-3b17bbeb28b6" containerName="istio-proxy" Nov 04 01:28:52.122361 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121482 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="899e8d41-45fd-4178-95d8-271b97510499" containerName="istio-proxy" Nov 04 01:28:52.122361 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121493 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ca1b1e-0ca9-485f-8f2b-1d013b618b66" containerName="istio-proxy" Nov 04 01:28:52.122361 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121504 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" Nov 04 01:28:52.122361 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.121512 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4447b9a-810f-4410-909c-9d51764eff35" containerName="maas-api" Nov 04 01:28:52.127374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.127354 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.179374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.179343 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-3dc6c8-serving-cert\"" Nov 04 01:28:52.186920 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.186878 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k"] Nov 04 01:28:52.226561 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.226527 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhlz\" (UniqueName: \"kubernetes.io/projected/f5182da9-f486-453f-80b3-2946aae607fa-kube-api-access-vwhlz\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.226758 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.226680 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5182da9-f486-453f-80b3-2946aae607fa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.226758 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.226750 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.226985 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.226969 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.328336 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.328297 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.328543 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.328381 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhlz\" (UniqueName: \"kubernetes.io/projected/f5182da9-f486-453f-80b3-2946aae607fa-kube-api-access-vwhlz\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.328543 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:52.328464 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-serving-cert: secret "maas-api-e2e-shared-b-w6-3dc6c8-serving-cert" not found Nov 04 01:28:52.328543 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.328474 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5182da9-f486-453f-80b3-2946aae607fa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.328679 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:28:52.328543 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-maas-api-tls podName:f5182da9-f486-453f-80b3-2946aae607fa nodeName:}" failed. No retries permitted until 2026-11-04 01:28:52.828521525 +0000 UTC m=+1878.444867266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-maas-api-tls") pod "maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" (UID: "f5182da9-f486-453f-80b3-2946aae607fa") : secret "maas-api-e2e-shared-b-w6-3dc6c8-serving-cert" not found Nov 04 01:28:52.328679 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.328592 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.329178 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.329148 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5182da9-f486-453f-80b3-2946aae607fa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.331000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.330980 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.419196 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.407877 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhlz\" (UniqueName: \"kubernetes.io/projected/f5182da9-f486-453f-80b3-2946aae607fa-kube-api-access-vwhlz\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.506132 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.506092 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4447b9a-810f-4410-909c-9d51764eff35" path="/var/lib/kubelet/pods/b4447b9a-810f-4410-909c-9d51764eff35/volumes" Nov 04 01:28:52.836564 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.836461 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:52.838936 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:52.838910 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:53.042441 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:53.042405 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:53.427519 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:53.427444 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k"] Nov 04 01:28:53.431332 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:28:53.431300 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5182da9_f486_453f_80b3_2946aae607fa.slice/crio-90c48b656a09e882f89959d5348b7a4fe484afe10a3eae00df7dea77f871550e WatchSource:0}: Error finding container 90c48b656a09e882f89959d5348b7a4fe484afe10a3eae00df7dea77f871550e: Status 404 returned error can't find the container with id 90c48b656a09e882f89959d5348b7a4fe484afe10a3eae00df7dea77f871550e Nov 04 01:28:53.433039 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:53.433022 5786 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:28:53.531153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:53.531088 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" event={"ID":"f5182da9-f486-453f-80b3-2946aae607fa","Type":"ContainerStarted","Data":"90c48b656a09e882f89959d5348b7a4fe484afe10a3eae00df7dea77f871550e"} Nov 04 01:28:54.539682 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:54.539645 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" event={"ID":"f5182da9-f486-453f-80b3-2946aae607fa","Type":"ContainerStarted","Data":"5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb"} Nov 04 01:28:54.540063 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:54.539819 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:28:54.674228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:28:54.674142 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" podStartSLOduration=1.675839307 podStartE2EDuration="2.674114154s" totalImagesPullingTime="998.274847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:54.673355604 +0000 UTC m=+1880.289701368" watchObservedRunningTime="2026-11-04 01:28:54.674114154 +0000 UTC m=+1880.290459917" Nov 04 01:29:00.568389 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:00.568262 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:29:04.872014 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:04.871977 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:29:06.703803 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:06.703768 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9"] Nov 04 01:29:08.597949 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.597917 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq"] Nov 04 01:29:08.598788 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.598765 5786 cpu_manager.go:414] "containerMap: removing container" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" Nov 04 01:29:08.598788 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.598787 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51822598-8499-472e-b731-4573487376e9" containerName="istio-proxy" Nov 04 01:29:08.603410 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.603387 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" Nov 04 01:29:08.919203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.918379 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-98f846-dockercfg-2qjxv\"" Nov 04 01:29:08.919203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.918552 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-98f846\"" Nov 04 01:29:08.931030 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.930994 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq\" (UID: \"fcd9752e-6a9b-470f-ad9d-938bca9a46cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" Nov 04 01:29:08.931385 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.931363 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8464\" (UniqueName: \"kubernetes.io/projected/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-kube-api-access-l8464\") pod \"payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq\" (UID: \"fcd9752e-6a9b-470f-ad9d-938bca9a46cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" Nov 04 01:29:08.933636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.931143 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq"] Nov 04 01:29:08.938438 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.938340 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c"] Nov 04 01:29:08.948286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.948263 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c"] Nov 04 01:29:08.948563 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.948552 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk"] Nov 04 01:29:08.951373 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.948506 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" Nov 04 01:29:08.954368 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:08.954350 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.020367 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.020334 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-5778ad-serving-cert\"" Nov 04 01:29:09.025307 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.025277 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk"] Nov 04 01:29:09.034719 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.034682 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8464\" (UniqueName: \"kubernetes.io/projected/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-kube-api-access-l8464\") pod \"payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq\" (UID: \"fcd9752e-6a9b-470f-ad9d-938bca9a46cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" Nov 04 01:29:09.034891 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.034749 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.034891 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.034788 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.035189 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.035151 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qqhb\" (UniqueName: \"kubernetes.io/projected/f10046d1-f25f-4ccd-9139-1f2a864d84e7-kube-api-access-6qqhb\") pod \"payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c\" (UID: \"f10046d1-f25f-4ccd-9139-1f2a864d84e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" Nov 04 01:29:09.035318 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.035208 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgx6m\" (UniqueName: \"kubernetes.io/projected/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-kube-api-access-xgx6m\") pod \"maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.035394 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.035378 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10046d1-f25f-4ccd-9139-1f2a864d84e7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c\" (UID: \"f10046d1-f25f-4ccd-9139-1f2a864d84e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" Nov 04 01:29:09.035565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.035545 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.035655 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.035602 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq\" (UID: \"fcd9752e-6a9b-470f-ad9d-938bca9a46cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" Nov 04 01:29:09.036688 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.036661 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq\" (UID: \"fcd9752e-6a9b-470f-ad9d-938bca9a46cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" Nov 04 01:29:09.091794 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.091755 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8464\" (UniqueName: \"kubernetes.io/projected/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-kube-api-access-l8464\") pod \"payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq\" (UID: \"fcd9752e-6a9b-470f-ad9d-938bca9a46cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" Nov 04 01:29:09.153144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.145305 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.153570 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.153537 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.162776 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.162704 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.163137 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.163092 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qqhb\" (UniqueName: \"kubernetes.io/projected/f10046d1-f25f-4ccd-9139-1f2a864d84e7-kube-api-access-6qqhb\") pod \"payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c\" (UID: \"f10046d1-f25f-4ccd-9139-1f2a864d84e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" Nov 04 01:29:09.167138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.164190 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgx6m\" (UniqueName: \"kubernetes.io/projected/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-kube-api-access-xgx6m\") pod \"maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.167138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.164283 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10046d1-f25f-4ccd-9139-1f2a864d84e7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c\" (UID: \"f10046d1-f25f-4ccd-9139-1f2a864d84e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" Nov 04 01:29:09.167138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.165446 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10046d1-f25f-4ccd-9139-1f2a864d84e7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c\" (UID: \"f10046d1-f25f-4ccd-9139-1f2a864d84e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" Nov 04 01:29:09.167138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.166143 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.180507 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.180433 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.187792 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.187761 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.216878 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.216846 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" Nov 04 01:29:09.238550 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.238516 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qqhb\" (UniqueName: \"kubernetes.io/projected/f10046d1-f25f-4ccd-9139-1f2a864d84e7-kube-api-access-6qqhb\") pod \"payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c\" (UID: \"f10046d1-f25f-4ccd-9139-1f2a864d84e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" Nov 04 01:29:09.238679 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.238654 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgx6m\" (UniqueName: \"kubernetes.io/projected/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-kube-api-access-xgx6m\") pod \"maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.274671 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.273595 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" Nov 04 01:29:09.278844 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.278421 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:09.593783 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.593742 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq"] Nov 04 01:29:09.598684 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:29:09.598649 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd9752e_6a9b_470f_ad9d_938bca9a46cc.slice/crio-c0d56a3388aa8e52b71f2f2ec01e88c7ea82eb4ca9b79412868513032782c5b7 WatchSource:0}: Error finding container c0d56a3388aa8e52b71f2f2ec01e88c7ea82eb4ca9b79412868513032782c5b7: Status 404 returned error can't find the container with id c0d56a3388aa8e52b71f2f2ec01e88c7ea82eb4ca9b79412868513032782c5b7 Nov 04 01:29:09.674456 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.674422 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" event={"ID":"fcd9752e-6a9b-470f-ad9d-938bca9a46cc","Type":"ContainerStarted","Data":"d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4"} Nov 04 01:29:09.674585 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.674489 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" event={"ID":"fcd9752e-6a9b-470f-ad9d-938bca9a46cc","Type":"ContainerStarted","Data":"c0d56a3388aa8e52b71f2f2ec01e88c7ea82eb4ca9b79412868513032782c5b7"} Nov 04 01:29:09.773751 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.773657 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk"] Nov 04 01:29:09.778307 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:29:09.778281 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb0ebe5_687e_4d22_a7c3_5ea8171acdc8.slice/crio-f316c94dc05c69457f8dcba83bb4131b1dcb20140ece693d44d13fa2f744203b WatchSource:0}: Error finding container f316c94dc05c69457f8dcba83bb4131b1dcb20140ece693d44d13fa2f744203b: Status 404 returned error can't find the container with id f316c94dc05c69457f8dcba83bb4131b1dcb20140ece693d44d13fa2f744203b Nov 04 01:29:09.822839 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:09.822802 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c"] Nov 04 01:29:09.827748 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:29:09.827718 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10046d1_f25f_4ccd_9139_1f2a864d84e7.slice/crio-2d32972cba6f29acbf44c8185b3828bb236b886d9ecd94e10456970bab1602aa WatchSource:0}: Error finding container 2d32972cba6f29acbf44c8185b3828bb236b886d9ecd94e10456970bab1602aa: Status 404 returned error can't find the container with id 2d32972cba6f29acbf44c8185b3828bb236b886d9ecd94e10456970bab1602aa Nov 04 01:29:10.682367 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:10.682340 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" event={"ID":"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8","Type":"ContainerStarted","Data":"f316c94dc05c69457f8dcba83bb4131b1dcb20140ece693d44d13fa2f744203b"} Nov 04 01:29:10.684165 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:10.684133 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" event={"ID":"f10046d1-f25f-4ccd-9139-1f2a864d84e7","Type":"ContainerStarted","Data":"a734d73eb64e42a76216b0f4a80cb4837e7d012301ad4bcc326e4a9515cc919e"} Nov 04 01:29:10.684285 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:10.684189 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" event={"ID":"f10046d1-f25f-4ccd-9139-1f2a864d84e7","Type":"ContainerStarted","Data":"2d32972cba6f29acbf44c8185b3828bb236b886d9ecd94e10456970bab1602aa"} Nov 04 01:29:10.684469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:10.684451 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" Nov 04 01:29:10.795008 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:10.794955 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" podStartSLOduration=2.794940963 podStartE2EDuration="2.794940963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:10.794623859 +0000 UTC m=+1896.410969622" watchObservedRunningTime="2026-11-04 01:29:10.794940963 +0000 UTC m=+1896.411286726" Nov 04 01:29:10.906430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:10.906374 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" podStartSLOduration=2.906358397 podStartE2EDuration="2.906358397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:10.905035903 +0000 UTC m=+1896.521381669" watchObservedRunningTime="2026-11-04 01:29:10.906358397 +0000 UTC m=+1896.522704159" Nov 04 01:29:11.692939 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:11.692904 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" event={"ID":"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8","Type":"ContainerStarted","Data":"e50175a88b8985791445f150e32ba2bd63a0af56d877eafe35fb2a92bb5f1f6a"} Nov 04 01:29:11.693469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:11.692991 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:11.693469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:11.693243 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" Nov 04 01:29:11.807390 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:11.807292 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" podStartSLOduration=2.924984373 podStartE2EDuration="3.807274133s" totalImagesPullingTime="882.28976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:11.804263713 +0000 UTC m=+1897.420609476" watchObservedRunningTime="2026-11-04 01:29:11.807274133 +0000 UTC m=+1897.423619896" Nov 04 01:29:14.167622 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:14.167535 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv"] Nov 04 01:29:15.089265 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:29:15.089223 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:29:17.705772 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:17.705744 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:29:19.167899 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.167833 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" podUID="41127884-4d48-4b30-a227-2a34a270c863" containerName="payload-processing" containerID="cri-o://3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c" gracePeriod=25 Nov 04 01:29:19.510873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.510848 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" Nov 04 01:29:19.578941 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.578908 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41127884-4d48-4b30-a227-2a34a270c863-kube-api-access-smxwz\" (UniqueName: \"kubernetes.io/projected/41127884-4d48-4b30-a227-2a34a270c863-kube-api-access-smxwz\") pod \"41127884-4d48-4b30-a227-2a34a270c863\" (UID: \"41127884-4d48-4b30-a227-2a34a270c863\") " Nov 04 01:29:19.579204 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.579187 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41127884-4d48-4b30-a227-2a34a270c863-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41127884-4d48-4b30-a227-2a34a270c863-plugins-config-volume\") pod \"41127884-4d48-4b30-a227-2a34a270c863\" (UID: \"41127884-4d48-4b30-a227-2a34a270c863\") " Nov 04 01:29:19.579576 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.579548 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41127884-4d48-4b30-a227-2a34a270c863-plugins-config-volume" pod "41127884-4d48-4b30-a227-2a34a270c863" (UID: "41127884-4d48-4b30-a227-2a34a270c863"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:19.579957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.579935 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41127884-4d48-4b30-a227-2a34a270c863-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:19.581208 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.581184 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41127884-4d48-4b30-a227-2a34a270c863-kube-api-access-smxwz" pod "41127884-4d48-4b30-a227-2a34a270c863" (UID: "41127884-4d48-4b30-a227-2a34a270c863"). InnerVolumeSpecName "kube-api-access-smxwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:19.681139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.681093 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smxwz\" (UniqueName: \"kubernetes.io/projected/41127884-4d48-4b30-a227-2a34a270c863-kube-api-access-smxwz\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:19.762605 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.762513 5786 generic.go:356] "Generic (PLEG): container finished" podID="41127884-4d48-4b30-a227-2a34a270c863" containerID="3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c" exitCode=0 Nov 04 01:29:19.762605 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.762564 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" Nov 04 01:29:19.762605 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.762588 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" event={"ID":"41127884-4d48-4b30-a227-2a34a270c863","Type":"ContainerDied","Data":"3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c"} Nov 04 01:29:19.762848 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.762640 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv" event={"ID":"41127884-4d48-4b30-a227-2a34a270c863","Type":"ContainerDied","Data":"d98996bf48846555919d9ab02daca0ec1c8f68a1388b0cd425931da45a0f53c2"} Nov 04 01:29:19.762848 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.762662 5786 scope.go:122] "RemoveContainer" containerID="3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c" Nov 04 01:29:19.773310 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.773295 5786 scope.go:122] "RemoveContainer" containerID="3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c" Nov 04 01:29:19.773524 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:29:19.773505 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c\": container with ID starting with 3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c not found: ID does not exist" containerID="3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c" Nov 04 01:29:19.773563 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.773532 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c"} err="failed to get container status \"3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c\": rpc error: code = NotFound desc = could not find container \"3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c\": container with ID starting with 3fb26c9c76b93204718595d01406fa18d8be20a4408c72e7b958c8995f45724c not found: ID does not exist" Nov 04 01:29:19.922839 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.922801 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv"] Nov 04 01:29:19.927872 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:19.927845 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e65b23-8599d6ff96-bwxrv"] Nov 04 01:29:20.506896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:20.506871 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41127884-4d48-4b30-a227-2a34a270c863" path="/var/lib/kubelet/pods/41127884-4d48-4b30-a227-2a34a270c863/volumes" Nov 04 01:29:21.696699 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:21.696670 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" Nov 04 01:29:21.725891 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:21.725802 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" containerID="cri-o://efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331" gracePeriod=45 Nov 04 01:29:22.027455 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.027430 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:29:22.115925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.115892 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-home\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-home\") pod \"388313a1-0083-4499-96eb-d8acd96280ed\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " Nov 04 01:29:22.116149 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.115996 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-model-cache\") pod \"388313a1-0083-4499-96eb-d8acd96280ed\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " Nov 04 01:29:22.116149 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.116060 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-dshm\") pod \"388313a1-0083-4499-96eb-d8acd96280ed\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " Nov 04 01:29:22.116289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.116198 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-tmp-dir\") pod \"388313a1-0083-4499-96eb-d8acd96280ed\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " Nov 04 01:29:22.116289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.116242 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-home" pod "388313a1-0083-4499-96eb-d8acd96280ed" (UID: "388313a1-0083-4499-96eb-d8acd96280ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:22.116289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.116278 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/388313a1-0083-4499-96eb-d8acd96280ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/388313a1-0083-4499-96eb-d8acd96280ed-tls-certs\") pod \"388313a1-0083-4499-96eb-d8acd96280ed\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " Nov 04 01:29:22.116414 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.116325 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-model-cache" pod "388313a1-0083-4499-96eb-d8acd96280ed" (UID: "388313a1-0083-4499-96eb-d8acd96280ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:22.116414 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.116367 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388313a1-0083-4499-96eb-d8acd96280ed-kube-api-access-84fxv\" (UniqueName: \"kubernetes.io/projected/388313a1-0083-4499-96eb-d8acd96280ed-kube-api-access-84fxv\") pod \"388313a1-0083-4499-96eb-d8acd96280ed\" (UID: \"388313a1-0083-4499-96eb-d8acd96280ed\") " Nov 04 01:29:22.116414 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.116393 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-tmp-dir" pod "388313a1-0083-4499-96eb-d8acd96280ed" (UID: "388313a1-0083-4499-96eb-d8acd96280ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:22.116808 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.116787 5786 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-home\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.116808 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.116804 5786 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-model-cache\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.116973 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.116814 5786 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-tmp-dir\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.118353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.118329 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388313a1-0083-4499-96eb-d8acd96280ed-tls-certs" pod "388313a1-0083-4499-96eb-d8acd96280ed" (UID: "388313a1-0083-4499-96eb-d8acd96280ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:29:22.118477 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.118389 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388313a1-0083-4499-96eb-d8acd96280ed-kube-api-access-84fxv" pod "388313a1-0083-4499-96eb-d8acd96280ed" (UID: "388313a1-0083-4499-96eb-d8acd96280ed"). InnerVolumeSpecName "kube-api-access-84fxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:22.118542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.118468 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-dshm" pod "388313a1-0083-4499-96eb-d8acd96280ed" (UID: "388313a1-0083-4499-96eb-d8acd96280ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:22.217978 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.217926 5786 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388313a1-0083-4499-96eb-d8acd96280ed-dshm\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.217978 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.217962 5786 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388313a1-0083-4499-96eb-d8acd96280ed-tls-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.217978 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.217979 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84fxv\" (UniqueName: \"kubernetes.io/projected/388313a1-0083-4499-96eb-d8acd96280ed-kube-api-access-84fxv\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:22.704047 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.704008 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" Nov 04 01:29:22.783300 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.783212 5786 generic.go:356] "Generic (PLEG): container finished" podID="388313a1-0083-4499-96eb-d8acd96280ed" containerID="efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331" exitCode=0 Nov 04 01:29:22.783300 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.783264 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" Nov 04 01:29:22.783494 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.783297 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" event={"ID":"388313a1-0083-4499-96eb-d8acd96280ed","Type":"ContainerDied","Data":"efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331"} Nov 04 01:29:22.783494 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.783343 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9" event={"ID":"388313a1-0083-4499-96eb-d8acd96280ed","Type":"ContainerDied","Data":"fefd30c299d30f049290b0ff3fbcc16b77aa32f33450b5b785321a2bbdae3c88"} Nov 04 01:29:22.783494 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.783360 5786 scope.go:122] "RemoveContainer" containerID="efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331" Nov 04 01:29:22.794491 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.794470 5786 scope.go:122] "RemoveContainer" containerID="efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331" Nov 04 01:29:22.795005 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:29:22.794972 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331\": container with ID starting with efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331 not found: ID does not exist" containerID="efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331" Nov 04 01:29:22.795149 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.795011 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331"} err="failed to get container status \"efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331\": rpc error: code = NotFound desc = could not find container \"efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331\": container with ID starting with efdb987ccc28d23f1f7c7ccb2750fa788b86ddc011a79b3ee6373dc12033d331 not found: ID does not exist" Nov 04 01:29:22.927741 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.927705 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9"] Nov 04 01:29:22.930373 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:22.930344 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e65b23/distinct-2-w4-e65b23-kserve-777f47b6b9-lqfp9"] Nov 04 01:29:24.023296 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.023256 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9"] Nov 04 01:29:24.024160 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.024133 5786 cpu_manager.go:414] "containerMap: removing container" podUID="41127884-4d48-4b30-a227-2a34a270c863" containerName="payload-processing" Nov 04 01:29:24.024160 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.024158 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41127884-4d48-4b30-a227-2a34a270c863" containerName="payload-processing" Nov 04 01:29:24.024315 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.024188 5786 cpu_manager.go:414] "containerMap: removing container" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" Nov 04 01:29:24.024315 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.024195 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" Nov 04 01:29:24.024315 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.024311 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="41127884-4d48-4b30-a227-2a34a270c863" containerName="payload-processing" Nov 04 01:29:24.024414 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.024324 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="388313a1-0083-4499-96eb-d8acd96280ed" containerName="main" Nov 04 01:29:24.029536 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.029511 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.081464 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.081432 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-42958c-serving-cert\"" Nov 04 01:29:24.134004 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.133962 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9"] Nov 04 01:29:24.143085 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.143047 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.143303 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.143110 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf329b96-b5f3-4c89-be83-0e107880e64f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.143371 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.143314 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.143457 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.143431 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpq8\" (UniqueName: \"kubernetes.io/projected/cf329b96-b5f3-4c89-be83-0e107880e64f-kube-api-access-kbpq8\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.244769 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.244734 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.249599 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.244812 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpq8\" (UniqueName: \"kubernetes.io/projected/cf329b96-b5f3-4c89-be83-0e107880e64f-kube-api-access-kbpq8\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.249599 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.245013 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.249599 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.245053 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf329b96-b5f3-4c89-be83-0e107880e64f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.249599 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:29:24.245968 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-serving-cert: secret "maas-api-e2e-shared-b-w0-42958c-serving-cert" not found Nov 04 01:29:24.249599 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:29:24.246051 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-maas-api-tls podName:cf329b96-b5f3-4c89-be83-0e107880e64f nodeName:}" failed. No retries permitted until 2026-11-04 01:29:24.746028286 +0000 UTC m=+1910.362374051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-maas-api-tls") pod "maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" (UID: "cf329b96-b5f3-4c89-be83-0e107880e64f") : secret "maas-api-e2e-shared-b-w0-42958c-serving-cert" not found Nov 04 01:29:24.249599 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.246898 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf329b96-b5f3-4c89-be83-0e107880e64f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.284343 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.279790 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.341142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.339698 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpq8\" (UniqueName: \"kubernetes.io/projected/cf329b96-b5f3-4c89-be83-0e107880e64f-kube-api-access-kbpq8\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.528026 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.527989 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388313a1-0083-4499-96eb-d8acd96280ed" path="/var/lib/kubelet/pods/388313a1-0083-4499-96eb-d8acd96280ed/volumes" Nov 04 01:29:24.761499 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.761463 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.764172 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.764142 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:24.944071 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:24.944040 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:25.348990 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:25.348617 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9"] Nov 04 01:29:25.354061 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:29:25.354033 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf329b96_b5f3_4c89_be83_0e107880e64f.slice/crio-d32d75a4ae77dab96176a4ea4e0fbf6def494ea5b853d4f241cc816511682cbe WatchSource:0}: Error finding container d32d75a4ae77dab96176a4ea4e0fbf6def494ea5b853d4f241cc816511682cbe: Status 404 returned error can't find the container with id d32d75a4ae77dab96176a4ea4e0fbf6def494ea5b853d4f241cc816511682cbe Nov 04 01:29:25.805926 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:25.805851 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" event={"ID":"cf329b96-b5f3-4c89-be83-0e107880e64f","Type":"ContainerStarted","Data":"d32d75a4ae77dab96176a4ea4e0fbf6def494ea5b853d4f241cc816511682cbe"} Nov 04 01:29:26.814706 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:26.814664 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" event={"ID":"cf329b96-b5f3-4c89-be83-0e107880e64f","Type":"ContainerStarted","Data":"9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62"} Nov 04 01:29:26.815142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:26.814735 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:26.936139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:26.936073 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" podStartSLOduration=3.024627711 podStartE2EDuration="3.936059479s" totalImagesPullingTime="911.431768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:26.934873497 +0000 UTC m=+1912.551219261" watchObservedRunningTime="2026-11-04 01:29:26.936059479 +0000 UTC m=+1912.552405242" Nov 04 01:29:32.826340 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:32.826309 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:29:41.147422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:41.147383 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:29:45.111067 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:29:45.111033 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:29:53.057622 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:53.057586 5786 generic.go:356] "Generic (PLEG): container finished" podID="3437edcf-befd-4454-b1db-f3ec59979527" containerID="0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8" exitCode=0 Nov 04 01:29:53.058062 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:53.057654 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" event={"ID":"3437edcf-befd-4454-b1db-f3ec59979527","Type":"ContainerDied","Data":"0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8"} Nov 04 01:29:53.058170 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:53.058110 5786 scope.go:122] "RemoveContainer" containerID="0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8" Nov 04 01:29:53.059799 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:53.059771 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:29:53.059878 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:53.059840 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:29:53.059919 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:53.059887 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:29:54.090186 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:54.089243 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" event={"ID":"3437edcf-befd-4454-b1db-f3ec59979527","Type":"ContainerStarted","Data":"86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445"} Nov 04 01:29:54.482150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:54.476825 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:29:54.482150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:54.477005 5786 patch_prober.go:29] interesting pod/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:29:54.482150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:54.477048 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" podUID="3437edcf-befd-4454-b1db-f3ec59979527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 04 01:29:55.480142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:55.479246 5786 patch_prober.go:29] interesting pod/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:29:55.480142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:55.479359 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" podUID="3437edcf-befd-4454-b1db-f3ec59979527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 04 01:29:56.487144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:56.478431 5786 patch_prober.go:29] interesting pod/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:29:56.487144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:56.478564 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" podUID="3437edcf-befd-4454-b1db-f3ec59979527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 04 01:29:56.828027 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:56.827939 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:29:57.398212 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.398177 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j"] Nov 04 01:29:57.401797 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.401777 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:57.454581 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.454544 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-98f846-serving-cert\"" Nov 04 01:29:57.488290 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.488257 5786 patch_prober.go:29] interesting pod/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:29:57.488868 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.488835 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" podUID="3437edcf-befd-4454-b1db-f3ec59979527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 04 01:29:57.505477 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.505443 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j"] Nov 04 01:29:57.559677 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.559640 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:57.559837 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.559708 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5mrh\" (UniqueName: \"kubernetes.io/projected/08f1f1d7-b68a-489c-8914-6525a4079c6a-kube-api-access-s5mrh\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:57.559837 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.559812 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:57.559913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.559858 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f1f1d7-b68a-489c-8914-6525a4079c6a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:57.661002 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.660907 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:57.661002 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.660973 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5mrh\" (UniqueName: \"kubernetes.io/projected/08f1f1d7-b68a-489c-8914-6525a4079c6a-kube-api-access-s5mrh\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:57.661298 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.661004 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:57.661298 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.661045 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f1f1d7-b68a-489c-8914-6525a4079c6a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:57.661298 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:29:57.661071 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-serving-cert: secret "maas-api-e2e-shared-a-w2-98f846-serving-cert" not found Nov 04 01:29:57.661298 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:29:57.661200 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-maas-api-tls podName:08f1f1d7-b68a-489c-8914-6525a4079c6a nodeName:}" failed. No retries permitted until 2026-11-04 01:29:58.161115135 +0000 UTC m=+1943.777460876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-maas-api-tls") pod "maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" (UID: "08f1f1d7-b68a-489c-8914-6525a4079c6a") : secret "maas-api-e2e-shared-a-w2-98f846-serving-cert" not found Nov 04 01:29:57.693144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.688797 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f1f1d7-b68a-489c-8914-6525a4079c6a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:57.719193 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.716579 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:57.732142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:57.726725 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5mrh\" (UniqueName: \"kubernetes.io/projected/08f1f1d7-b68a-489c-8914-6525a4079c6a-kube-api-access-s5mrh\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:58.201150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:58.197212 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:58.240180 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:58.239930 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:58.314834 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:58.314797 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:29:58.478708 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:58.478621 5786 patch_prober.go:29] interesting pod/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:29:58.486264 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:58.486222 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" podUID="3437edcf-befd-4454-b1db-f3ec59979527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 04 01:29:58.725897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:58.725857 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j"] Nov 04 01:29:58.731002 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:29:58.730944 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f1f1d7_b68a_489c_8914_6525a4079c6a.slice/crio-26114c987eaa47f1a7c51e1dab239d6183680b8cb944c5b2029e7c3fefeeb1c4 WatchSource:0}: Error finding container 26114c987eaa47f1a7c51e1dab239d6183680b8cb944c5b2029e7c3fefeeb1c4: Status 404 returned error can't find the container with id 26114c987eaa47f1a7c51e1dab239d6183680b8cb944c5b2029e7c3fefeeb1c4 Nov 04 01:29:59.171434 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:59.171389 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf"] Nov 04 01:29:59.171654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:59.171620 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" podUID="889517a5-9d9a-4643-9665-dd6b46496379" containerName="istio-proxy" containerID="cri-o://33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122" gracePeriod=30 Nov 04 01:29:59.180298 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:59.180268 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" event={"ID":"08f1f1d7-b68a-489c-8914-6525a4079c6a","Type":"ContainerStarted","Data":"26114c987eaa47f1a7c51e1dab239d6183680b8cb944c5b2029e7c3fefeeb1c4"} Nov 04 01:29:59.194222 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:59.194190 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc"] Nov 04 01:29:59.194431 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:59.194403 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" podUID="bb8d2819-e753-4b38-864b-49faf969feac" containerName="istio-proxy" containerID="cri-o://59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726" gracePeriod=30 Nov 04 01:29:59.200012 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:59.199983 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p"] Nov 04 01:29:59.200246 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:59.200222 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" containerID="cri-o://72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce" gracePeriod=30 Nov 04 01:29:59.568877 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:59.568801 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:29:59.570644 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:29:59.570618 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:30:00.188520 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.188485 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" event={"ID":"08f1f1d7-b68a-489c-8914-6525a4079c6a","Type":"ContainerStarted","Data":"9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd"} Nov 04 01:30:00.188983 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.188729 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:30:00.340633 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.340599 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d"] Nov 04 01:30:00.342349 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.342296 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" podStartSLOduration=2.429748178 podStartE2EDuration="3.342280568s" totalImagesPullingTime="912.53239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:00.333215002 +0000 UTC m=+1945.949560767" watchObservedRunningTime="2026-11-04 01:30:00.342280568 +0000 UTC m=+1945.958626333" Nov 04 01:30:00.344565 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.344541 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" Nov 04 01:30:00.406868 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.406828 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d"] Nov 04 01:30:00.451547 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.451474 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:30:00.452176 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.452148 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-924vg\" (UniqueName: \"kubernetes.io/projected/bb562218-5de4-425e-9226-2c3fe590fda9-kube-api-access-924vg\") pod \"maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d\" (UID: \"bb562218-5de4-425e-9226-2c3fe590fda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" Nov 04 01:30:00.452466 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.452445 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb562218-5de4-425e-9226-2c3fe590fda9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d\" (UID: \"bb562218-5de4-425e-9226-2c3fe590fda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" Nov 04 01:30:00.562308 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.562259 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb562218-5de4-425e-9226-2c3fe590fda9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d\" (UID: \"bb562218-5de4-425e-9226-2c3fe590fda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" Nov 04 01:30:00.562641 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.562610 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-924vg\" (UniqueName: \"kubernetes.io/projected/bb562218-5de4-425e-9226-2c3fe590fda9-kube-api-access-924vg\") pod \"maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d\" (UID: \"bb562218-5de4-425e-9226-2c3fe590fda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" Nov 04 01:30:00.563365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.563344 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb562218-5de4-425e-9226-2c3fe590fda9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d\" (UID: \"bb562218-5de4-425e-9226-2c3fe590fda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" Nov 04 01:30:00.619812 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.619773 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-924vg\" (UniqueName: \"kubernetes.io/projected/bb562218-5de4-425e-9226-2c3fe590fda9-kube-api-access-924vg\") pod \"maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d\" (UID: \"bb562218-5de4-425e-9226-2c3fe590fda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" Nov 04 01:30:00.660191 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:00.660158 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" Nov 04 01:30:01.037286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:01.037183 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d"] Nov 04 01:30:01.041530 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:30:01.041496 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb562218_5de4_425e_9226_2c3fe590fda9.slice/crio-4635c3e36d06f6b7458adea815551cb372ce5a3b8071fc33e153ab36370deae2 WatchSource:0}: Error finding container 4635c3e36d06f6b7458adea815551cb372ce5a3b8071fc33e153ab36370deae2: Status 404 returned error can't find the container with id 4635c3e36d06f6b7458adea815551cb372ce5a3b8071fc33e153ab36370deae2 Nov 04 01:30:01.195910 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:01.195874 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" event={"ID":"bb562218-5de4-425e-9226-2c3fe590fda9","Type":"ContainerStarted","Data":"4635c3e36d06f6b7458adea815551cb372ce5a3b8071fc33e153ab36370deae2"} Nov 04 01:30:02.204316 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:02.204277 5786 generic.go:356] "Generic (PLEG): container finished" podID="bb562218-5de4-425e-9226-2c3fe590fda9" containerID="459fb59c86a3a1c84faa2dd91f98acb405181c0f7c7b296b1b29faf4acb03f89" exitCode=60 Nov 04 01:30:02.204713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:02.204356 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" event={"ID":"bb562218-5de4-425e-9226-2c3fe590fda9","Type":"ContainerDied","Data":"459fb59c86a3a1c84faa2dd91f98acb405181c0f7c7b296b1b29faf4acb03f89"} Nov 04 01:30:02.204713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:02.204615 5786 scope.go:122] "RemoveContainer" containerID="459fb59c86a3a1c84faa2dd91f98acb405181c0f7c7b296b1b29faf4acb03f89" Nov 04 01:30:03.213634 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:03.213599 5786 generic.go:356] "Generic (PLEG): container finished" podID="bb562218-5de4-425e-9226-2c3fe590fda9" containerID="b3195220a8e92a46681240731a1beb4f47abe0cd4a4898b1bc3d066d56de085b" exitCode=60 Nov 04 01:30:03.214050 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:03.213680 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" event={"ID":"bb562218-5de4-425e-9226-2c3fe590fda9","Type":"ContainerDied","Data":"b3195220a8e92a46681240731a1beb4f47abe0cd4a4898b1bc3d066d56de085b"} Nov 04 01:30:03.214050 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:03.213734 5786 scope.go:122] "RemoveContainer" containerID="459fb59c86a3a1c84faa2dd91f98acb405181c0f7c7b296b1b29faf4acb03f89" Nov 04 01:30:03.214050 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:03.213945 5786 scope.go:122] "RemoveContainer" containerID="b3195220a8e92a46681240731a1beb4f47abe0cd4a4898b1bc3d066d56de085b" Nov 04 01:30:03.214250 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:30:03.214230 5786 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d_odh-ai-gateway-infra(bb562218-5de4-425e-9226-2c3fe590fda9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" Nov 04 01:30:04.221892 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.221867 5786 scope.go:122] "RemoveContainer" containerID="b3195220a8e92a46681240731a1beb4f47abe0cd4a4898b1bc3d066d56de085b" Nov 04 01:30:04.222290 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:30:04.222215 5786 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d_odh-ai-gateway-infra(bb562218-5de4-425e-9226-2c3fe590fda9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" Nov 04 01:30:04.480424 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.480394 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:30:04.530770 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.530740 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-kube-api-access-pdq4n\" (UniqueName: \"kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-kube-api-access-pdq4n\") pod \"889517a5-9d9a-4643-9665-dd6b46496379\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " Nov 04 01:30:04.530925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.530795 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-envoy\") pod \"889517a5-9d9a-4643-9665-dd6b46496379\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " Nov 04 01:30:04.530925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.530828 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/889517a5-9d9a-4643-9665-dd6b46496379-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889517a5-9d9a-4643-9665-dd6b46496379-istiod-ca-cert\") pod \"889517a5-9d9a-4643-9665-dd6b46496379\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " Nov 04 01:30:04.530925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.530856 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-credential-socket\") pod \"889517a5-9d9a-4643-9665-dd6b46496379\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " Nov 04 01:30:04.531112 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.531005 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-certs\") pod \"889517a5-9d9a-4643-9665-dd6b46496379\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " Nov 04 01:30:04.531112 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.531057 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/889517a5-9d9a-4643-9665-dd6b46496379-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889517a5-9d9a-4643-9665-dd6b46496379-istio-podinfo\") pod \"889517a5-9d9a-4643-9665-dd6b46496379\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " Nov 04 01:30:04.531374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.531241 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-istio-token\" (UniqueName: \"kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-istio-token\") pod \"889517a5-9d9a-4643-9665-dd6b46496379\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " Nov 04 01:30:04.531374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.531288 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-credential-socket" pod "889517a5-9d9a-4643-9665-dd6b46496379" (UID: "889517a5-9d9a-4643-9665-dd6b46496379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.531374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.531302 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889517a5-9d9a-4643-9665-dd6b46496379-istiod-ca-cert" pod "889517a5-9d9a-4643-9665-dd6b46496379" (UID: "889517a5-9d9a-4643-9665-dd6b46496379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:04.531374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.531314 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-data\") pod \"889517a5-9d9a-4643-9665-dd6b46496379\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " Nov 04 01:30:04.531374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.531370 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-socket\") pod \"889517a5-9d9a-4643-9665-dd6b46496379\" (UID: \"889517a5-9d9a-4643-9665-dd6b46496379\") " Nov 04 01:30:04.532951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.532060 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889517a5-9d9a-4643-9665-dd6b46496379-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.532951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.532087 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.532951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.532391 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-certs" pod "889517a5-9d9a-4643-9665-dd6b46496379" (UID: "889517a5-9d9a-4643-9665-dd6b46496379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.532951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.532866 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-socket" pod "889517a5-9d9a-4643-9665-dd6b46496379" (UID: "889517a5-9d9a-4643-9665-dd6b46496379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.532951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.532904 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-data" pod "889517a5-9d9a-4643-9665-dd6b46496379" (UID: "889517a5-9d9a-4643-9665-dd6b46496379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.533533 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.533506 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-kube-api-access-pdq4n" pod "889517a5-9d9a-4643-9665-dd6b46496379" (UID: "889517a5-9d9a-4643-9665-dd6b46496379"). InnerVolumeSpecName "kube-api-access-pdq4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:04.533725 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.533704 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/889517a5-9d9a-4643-9665-dd6b46496379-istio-podinfo" pod "889517a5-9d9a-4643-9665-dd6b46496379" (UID: "889517a5-9d9a-4643-9665-dd6b46496379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:04.534073 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.534061 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-istio-token" pod "889517a5-9d9a-4643-9665-dd6b46496379" (UID: "889517a5-9d9a-4643-9665-dd6b46496379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:04.535411 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.535378 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-envoy" pod "889517a5-9d9a-4643-9665-dd6b46496379" (UID: "889517a5-9d9a-4643-9665-dd6b46496379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.547986 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.547963 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:30:04.550035 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.550017 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:30:04.632840 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.632803 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-socket\") pod \"bb8d2819-e753-4b38-864b-49faf969feac\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " Nov 04 01:30:04.632999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.632890 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8d2819-e753-4b38-864b-49faf969feac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8d2819-e753-4b38-864b-49faf969feac-istiod-ca-cert\") pod \"bb8d2819-e753-4b38-864b-49faf969feac\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " Nov 04 01:30:04.632999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.632960 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-data\") pod \"bb8d2819-e753-4b38-864b-49faf969feac\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " Nov 04 01:30:04.632999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.632989 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-certs\") pod \"bb8d2819-e753-4b38-864b-49faf969feac\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " Nov 04 01:30:04.633211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633036 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-socket\") pod \"6b454222-560a-4d7d-8b07-997a983463c7\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " Nov 04 01:30:04.633211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633104 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b454222-560a-4d7d-8b07-997a983463c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b454222-560a-4d7d-8b07-997a983463c7-istio-podinfo\") pod \"6b454222-560a-4d7d-8b07-997a983463c7\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " Nov 04 01:30:04.633211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633162 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-socket" pod "bb8d2819-e753-4b38-864b-49faf969feac" (UID: "bb8d2819-e753-4b38-864b-49faf969feac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.633211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633175 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-data\") pod \"6b454222-560a-4d7d-8b07-997a983463c7\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " Nov 04 01:30:04.633419 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633303 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b454222-560a-4d7d-8b07-997a983463c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b454222-560a-4d7d-8b07-997a983463c7-istiod-ca-cert\") pod \"6b454222-560a-4d7d-8b07-997a983463c7\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " Nov 04 01:30:04.633419 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633331 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-data" pod "bb8d2819-e753-4b38-864b-49faf969feac" (UID: "bb8d2819-e753-4b38-864b-49faf969feac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.633419 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633341 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-istio-token\" (UniqueName: \"kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-istio-token\") pod \"6b454222-560a-4d7d-8b07-997a983463c7\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " Nov 04 01:30:04.633419 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633406 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8d2819-e753-4b38-864b-49faf969feac-istiod-ca-cert" pod "bb8d2819-e753-4b38-864b-49faf969feac" (UID: "bb8d2819-e753-4b38-864b-49faf969feac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:04.633636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633423 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-istio-token\" (UniqueName: \"kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-istio-token\") pod \"bb8d2819-e753-4b38-864b-49faf969feac\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " Nov 04 01:30:04.633636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633427 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-data" pod "6b454222-560a-4d7d-8b07-997a983463c7" (UID: "6b454222-560a-4d7d-8b07-997a983463c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.633636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633462 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-envoy\") pod \"bb8d2819-e753-4b38-864b-49faf969feac\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " Nov 04 01:30:04.633636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633526 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb8d2819-e753-4b38-864b-49faf969feac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8d2819-e753-4b38-864b-49faf969feac-istio-podinfo\") pod \"bb8d2819-e753-4b38-864b-49faf969feac\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " Nov 04 01:30:04.633636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633565 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-credential-socket\") pod \"6b454222-560a-4d7d-8b07-997a983463c7\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " Nov 04 01:30:04.633636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633597 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-certs\") pod \"6b454222-560a-4d7d-8b07-997a983463c7\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " Nov 04 01:30:04.633937 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633647 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-kube-api-access-cktlh\" (UniqueName: \"kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-kube-api-access-cktlh\") pod \"6b454222-560a-4d7d-8b07-997a983463c7\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " Nov 04 01:30:04.633937 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633651 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-certs" pod "bb8d2819-e753-4b38-864b-49faf969feac" (UID: "bb8d2819-e753-4b38-864b-49faf969feac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.633937 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633701 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-envoy\") pod \"6b454222-560a-4d7d-8b07-997a983463c7\" (UID: \"6b454222-560a-4d7d-8b07-997a983463c7\") " Nov 04 01:30:04.633937 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633785 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-kube-api-access-dnx9z\" (UniqueName: \"kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-kube-api-access-dnx9z\") pod \"bb8d2819-e753-4b38-864b-49faf969feac\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " Nov 04 01:30:04.633937 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633806 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-socket" pod "6b454222-560a-4d7d-8b07-997a983463c7" (UID: "6b454222-560a-4d7d-8b07-997a983463c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.633937 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633828 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b454222-560a-4d7d-8b07-997a983463c7-istiod-ca-cert" pod "6b454222-560a-4d7d-8b07-997a983463c7" (UID: "6b454222-560a-4d7d-8b07-997a983463c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:04.633937 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.633866 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-credential-socket\") pod \"bb8d2819-e753-4b38-864b-49faf969feac\" (UID: \"bb8d2819-e753-4b38-864b-49faf969feac\") " Nov 04 01:30:04.634347 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634100 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-credential-socket" pod "6b454222-560a-4d7d-8b07-997a983463c7" (UID: "6b454222-560a-4d7d-8b07-997a983463c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.634347 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634324 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-credential-socket" pod "bb8d2819-e753-4b38-864b-49faf969feac" (UID: "bb8d2819-e753-4b38-864b-49faf969feac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.634627 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634602 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634702 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634635 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634702 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634652 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634702 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634666 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb8d2819-e753-4b38-864b-49faf969feac-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634702 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634680 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634702 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634694 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634707 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634722 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634736 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634749 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889517a5-9d9a-4643-9665-dd6b46496379-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634763 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b454222-560a-4d7d-8b07-997a983463c7-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634776 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634789 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634801 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634813 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889517a5-9d9a-4643-9665-dd6b46496379-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.634979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.634826 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdq4n\" (UniqueName: \"kubernetes.io/projected/889517a5-9d9a-4643-9665-dd6b46496379-kube-api-access-pdq4n\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.637733 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.637539 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b454222-560a-4d7d-8b07-997a983463c7-istio-podinfo" pod "6b454222-560a-4d7d-8b07-997a983463c7" (UID: "6b454222-560a-4d7d-8b07-997a983463c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:04.637967 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.637848 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-istio-token" pod "6b454222-560a-4d7d-8b07-997a983463c7" (UID: "6b454222-560a-4d7d-8b07-997a983463c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:04.637967 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.637902 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-certs" pod "6b454222-560a-4d7d-8b07-997a983463c7" (UID: "6b454222-560a-4d7d-8b07-997a983463c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.638414 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.638391 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-envoy" pod "bb8d2819-e753-4b38-864b-49faf969feac" (UID: "bb8d2819-e753-4b38-864b-49faf969feac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.638658 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.638629 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb8d2819-e753-4b38-864b-49faf969feac-istio-podinfo" pod "bb8d2819-e753-4b38-864b-49faf969feac" (UID: "bb8d2819-e753-4b38-864b-49faf969feac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:04.638819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.638793 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-kube-api-access-cktlh" pod "6b454222-560a-4d7d-8b07-997a983463c7" (UID: "6b454222-560a-4d7d-8b07-997a983463c7"). InnerVolumeSpecName "kube-api-access-cktlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:04.638899 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.638828 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-envoy" pod "6b454222-560a-4d7d-8b07-997a983463c7" (UID: "6b454222-560a-4d7d-8b07-997a983463c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.638899 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.638877 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-istio-token" pod "bb8d2819-e753-4b38-864b-49faf969feac" (UID: "bb8d2819-e753-4b38-864b-49faf969feac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:04.639283 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.639264 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-kube-api-access-dnx9z" pod "bb8d2819-e753-4b38-864b-49faf969feac" (UID: "bb8d2819-e753-4b38-864b-49faf969feac"). InnerVolumeSpecName "kube-api-access-dnx9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:04.736527 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.736444 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.736527 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.736473 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb8d2819-e753-4b38-864b-49faf969feac-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.736527 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.736482 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb8d2819-e753-4b38-864b-49faf969feac-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.736527 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.736493 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.736527 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.736503 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cktlh\" (UniqueName: \"kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-kube-api-access-cktlh\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.736527 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.736511 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b454222-560a-4d7d-8b07-997a983463c7-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.736527 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.736520 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnx9z\" (UniqueName: \"kubernetes.io/projected/bb8d2819-e753-4b38-864b-49faf969feac-kube-api-access-dnx9z\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.736527 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.736529 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b454222-560a-4d7d-8b07-997a983463c7-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.736898 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:04.736538 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b454222-560a-4d7d-8b07-997a983463c7-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:05.230542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.230509 5786 generic.go:356] "Generic (PLEG): container finished" podID="889517a5-9d9a-4643-9665-dd6b46496379" containerID="33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122" exitCode=0 Nov 04 01:30:05.230950 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.230551 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" Nov 04 01:30:05.230950 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.230585 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" event={"ID":"889517a5-9d9a-4643-9665-dd6b46496379","Type":"ContainerDied","Data":"33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122"} Nov 04 01:30:05.230950 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.230645 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf" event={"ID":"889517a5-9d9a-4643-9665-dd6b46496379","Type":"ContainerDied","Data":"2e5300cefed94070172aac28875bfae223ae6ce7233b0126f84822d5736ca709"} Nov 04 01:30:05.230950 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.230674 5786 scope.go:122] "RemoveContainer" containerID="33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122" Nov 04 01:30:05.232211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.232190 5786 generic.go:356] "Generic (PLEG): container finished" podID="6b454222-560a-4d7d-8b07-997a983463c7" containerID="72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce" exitCode=0 Nov 04 01:30:05.232311 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.232233 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" event={"ID":"6b454222-560a-4d7d-8b07-997a983463c7","Type":"ContainerDied","Data":"72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce"} Nov 04 01:30:05.232311 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.232288 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" event={"ID":"6b454222-560a-4d7d-8b07-997a983463c7","Type":"ContainerDied","Data":"101c0b58ac4d664fde915b5dab6c1a097b84d2bbc89938d741d64b83208c568d"} Nov 04 01:30:05.232311 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.232242 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p" Nov 04 01:30:05.233619 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.233597 5786 generic.go:356] "Generic (PLEG): container finished" podID="bb8d2819-e753-4b38-864b-49faf969feac" containerID="59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726" exitCode=0 Nov 04 01:30:05.233713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.233641 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" Nov 04 01:30:05.233713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.233654 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" event={"ID":"bb8d2819-e753-4b38-864b-49faf969feac","Type":"ContainerDied","Data":"59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726"} Nov 04 01:30:05.233713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.233691 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc" event={"ID":"bb8d2819-e753-4b38-864b-49faf969feac","Type":"ContainerDied","Data":"45e77d2c7e2a3c4d1ad9cf7ae1d2b1b1f989e7d50e64f2aa310dea70c919a434"} Nov 04 01:30:05.241763 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.241746 5786 scope.go:122] "RemoveContainer" containerID="33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122" Nov 04 01:30:05.242327 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:30:05.242302 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122\": container with ID starting with 33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122 not found: ID does not exist" containerID="33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122" Nov 04 01:30:05.242468 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.242334 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122"} err="failed to get container status \"33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122\": rpc error: code = NotFound desc = could not find container \"33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122\": container with ID starting with 33e5b9af86195a501166f7c92a4bc1f5427b0e24a7c993d6517462c0e944e122 not found: ID does not exist" Nov 04 01:30:05.242468 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.242353 5786 scope.go:122] "RemoveContainer" containerID="72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce" Nov 04 01:30:05.253339 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.253315 5786 scope.go:122] "RemoveContainer" containerID="72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce" Nov 04 01:30:05.253659 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:30:05.253640 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce\": container with ID starting with 72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce not found: ID does not exist" containerID="72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce" Nov 04 01:30:05.253707 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.253667 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce"} err="failed to get container status \"72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce\": rpc error: code = NotFound desc = could not find container \"72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce\": container with ID starting with 72b1421988c72d3c99df1efcb2daaaa9b3694305970d4db3e01daead36f352ce not found: ID does not exist" Nov 04 01:30:05.253707 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.253684 5786 scope.go:122] "RemoveContainer" containerID="59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726" Nov 04 01:30:05.262851 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.262831 5786 scope.go:122] "RemoveContainer" containerID="59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726" Nov 04 01:30:05.263170 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:30:05.263150 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726\": container with ID starting with 59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726 not found: ID does not exist" containerID="59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726" Nov 04 01:30:05.263227 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.263177 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726"} err="failed to get container status \"59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726\": rpc error: code = NotFound desc = could not find container \"59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726\": container with ID starting with 59d1f5bf9cf8c2cfab48cb74ecff2dd0337b923ca4b2ab212b399d5521e8d726 not found: ID does not exist" Nov 04 01:30:05.390432 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.390389 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc"] Nov 04 01:30:05.438175 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.438109 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-s2lpc"] Nov 04 01:30:05.547455 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.547368 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf"] Nov 04 01:30:05.551918 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.551888 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9rfdf"] Nov 04 01:30:05.658089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.658047 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p"] Nov 04 01:30:05.666644 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:05.666617 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-l565p"] Nov 04 01:30:06.278202 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:06.267252 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:30:06.506154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:06.506100 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b454222-560a-4d7d-8b07-997a983463c7" path="/var/lib/kubelet/pods/6b454222-560a-4d7d-8b07-997a983463c7/volumes" Nov 04 01:30:06.506773 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:06.506750 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="889517a5-9d9a-4643-9665-dd6b46496379" path="/var/lib/kubelet/pods/889517a5-9d9a-4643-9665-dd6b46496379/volumes" Nov 04 01:30:06.507384 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:06.507363 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb8d2819-e753-4b38-864b-49faf969feac" path="/var/lib/kubelet/pods/bb8d2819-e753-4b38-864b-49faf969feac/volumes" Nov 04 01:30:14.763345 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:14.763311 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:30:15.089360 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:30:15.089267 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:30:15.509140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:15.498502 5786 scope.go:122] "RemoveContainer" containerID="b3195220a8e92a46681240731a1beb4f47abe0cd4a4898b1bc3d066d56de085b" Nov 04 01:30:16.387513 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:16.387480 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" event={"ID":"bb562218-5de4-425e-9226-2c3fe590fda9","Type":"ContainerStarted","Data":"ce27f04bd397db5651b4ee8d23c760d8c43dafcefd1d7a94d3b0d6a25a619fce"} Nov 04 01:30:16.501230 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:16.501178 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" podStartSLOduration=16.501164308 podStartE2EDuration="16.501164308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:16.498517385 +0000 UTC m=+1962.114863150" watchObservedRunningTime="2026-11-04 01:30:16.501164308 +0000 UTC m=+1962.117510071" Nov 04 01:30:17.396106 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:17.396073 5786 generic.go:356] "Generic (PLEG): container finished" podID="bb562218-5de4-425e-9226-2c3fe590fda9" containerID="ce27f04bd397db5651b4ee8d23c760d8c43dafcefd1d7a94d3b0d6a25a619fce" exitCode=60 Nov 04 01:30:17.396516 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:17.396146 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" event={"ID":"bb562218-5de4-425e-9226-2c3fe590fda9","Type":"ContainerDied","Data":"ce27f04bd397db5651b4ee8d23c760d8c43dafcefd1d7a94d3b0d6a25a619fce"} Nov 04 01:30:17.396516 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:17.396209 5786 scope.go:122] "RemoveContainer" containerID="b3195220a8e92a46681240731a1beb4f47abe0cd4a4898b1bc3d066d56de085b" Nov 04 01:30:17.396613 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:17.396566 5786 scope.go:122] "RemoveContainer" containerID="ce27f04bd397db5651b4ee8d23c760d8c43dafcefd1d7a94d3b0d6a25a619fce" Nov 04 01:30:17.396914 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:30:17.396892 5786 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d_odh-ai-gateway-infra(bb562218-5de4-425e-9226-2c3fe590fda9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" Nov 04 01:30:17.537262 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:17.537225 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d"] Nov 04 01:30:18.554065 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:18.554041 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" Nov 04 01:30:18.655551 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:18.655474 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb562218-5de4-425e-9226-2c3fe590fda9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb562218-5de4-425e-9226-2c3fe590fda9-maas-api-service-ca\") pod \"bb562218-5de4-425e-9226-2c3fe590fda9\" (UID: \"bb562218-5de4-425e-9226-2c3fe590fda9\") " Nov 04 01:30:18.655551 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:18.655533 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb562218-5de4-425e-9226-2c3fe590fda9-kube-api-access-924vg\" (UniqueName: \"kubernetes.io/projected/bb562218-5de4-425e-9226-2c3fe590fda9-kube-api-access-924vg\") pod \"bb562218-5de4-425e-9226-2c3fe590fda9\" (UID: \"bb562218-5de4-425e-9226-2c3fe590fda9\") " Nov 04 01:30:18.655820 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:18.655795 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb562218-5de4-425e-9226-2c3fe590fda9-maas-api-service-ca" pod "bb562218-5de4-425e-9226-2c3fe590fda9" (UID: "bb562218-5de4-425e-9226-2c3fe590fda9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:18.656052 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:18.656032 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb562218-5de4-425e-9226-2c3fe590fda9-maas-api-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:18.657546 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:18.657525 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb562218-5de4-425e-9226-2c3fe590fda9-kube-api-access-924vg" pod "bb562218-5de4-425e-9226-2c3fe590fda9" (UID: "bb562218-5de4-425e-9226-2c3fe590fda9"). InnerVolumeSpecName "kube-api-access-924vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:18.757410 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:18.757378 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-924vg\" (UniqueName: \"kubernetes.io/projected/bb562218-5de4-425e-9226-2c3fe590fda9-kube-api-access-924vg\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:18.902502 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:18.902464 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj"] Nov 04 01:30:19.001093 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:19.000992 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d"] Nov 04 01:30:19.413930 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:19.413902 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" Nov 04 01:30:19.414148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:19.413930 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d" event={"ID":"bb562218-5de4-425e-9226-2c3fe590fda9","Type":"ContainerDied","Data":"4635c3e36d06f6b7458adea815551cb372ce5a3b8071fc33e153ab36370deae2"} Nov 04 01:30:19.414255 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:19.414197 5786 scope.go:122] "RemoveContainer" containerID="ce27f04bd397db5651b4ee8d23c760d8c43dafcefd1d7a94d3b0d6a25a619fce" Nov 04 01:30:19.608523 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:19.608488 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d"] Nov 04 01:30:19.611474 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:19.611449 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e210ef-29895930-cf82d"] Nov 04 01:30:20.550142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:20.548993 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" path="/var/lib/kubelet/pods/bb562218-5de4-425e-9226-2c3fe590fda9/volumes" Nov 04 01:30:23.902881 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:23.902845 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" podUID="3405d103-6146-4e19-9055-106154b2c39c" containerName="payload-processing" containerID="cri-o://86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900" gracePeriod=25 Nov 04 01:30:24.001827 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.001777 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" podUID="e3e67e26-14bd-477e-b842-192972b302ab" containerName="payload-pre-processing" containerID="cri-o://ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25" gracePeriod=25 Nov 04 01:30:24.196330 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.196298 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" Nov 04 01:30:24.236479 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.236443 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3405d103-6146-4e19-9055-106154b2c39c-kube-api-access-nj4ls\" (UniqueName: \"kubernetes.io/projected/3405d103-6146-4e19-9055-106154b2c39c-kube-api-access-nj4ls\") pod \"3405d103-6146-4e19-9055-106154b2c39c\" (UID: \"3405d103-6146-4e19-9055-106154b2c39c\") " Nov 04 01:30:24.236831 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.236812 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3405d103-6146-4e19-9055-106154b2c39c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3405d103-6146-4e19-9055-106154b2c39c-plugins-config-volume\") pod \"3405d103-6146-4e19-9055-106154b2c39c\" (UID: \"3405d103-6146-4e19-9055-106154b2c39c\") " Nov 04 01:30:24.237255 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.237217 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3405d103-6146-4e19-9055-106154b2c39c-plugins-config-volume" pod "3405d103-6146-4e19-9055-106154b2c39c" (UID: "3405d103-6146-4e19-9055-106154b2c39c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:24.237413 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.237394 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3405d103-6146-4e19-9055-106154b2c39c-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:24.243330 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.243299 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3405d103-6146-4e19-9055-106154b2c39c-kube-api-access-nj4ls" pod "3405d103-6146-4e19-9055-106154b2c39c" (UID: "3405d103-6146-4e19-9055-106154b2c39c"). InnerVolumeSpecName "kube-api-access-nj4ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:24.259026 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.259006 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" Nov 04 01:30:24.338172 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.338051 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3e67e26-14bd-477e-b842-192972b302ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e67e26-14bd-477e-b842-192972b302ab-plugins-config-volume\") pod \"e3e67e26-14bd-477e-b842-192972b302ab\" (UID: \"e3e67e26-14bd-477e-b842-192972b302ab\") " Nov 04 01:30:24.338343 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.338215 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e67e26-14bd-477e-b842-192972b302ab-kube-api-access-lc6mq\" (UniqueName: \"kubernetes.io/projected/e3e67e26-14bd-477e-b842-192972b302ab-kube-api-access-lc6mq\") pod \"e3e67e26-14bd-477e-b842-192972b302ab\" (UID: \"e3e67e26-14bd-477e-b842-192972b302ab\") " Nov 04 01:30:24.338986 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.338785 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e67e26-14bd-477e-b842-192972b302ab-plugins-config-volume" pod "e3e67e26-14bd-477e-b842-192972b302ab" (UID: "e3e67e26-14bd-477e-b842-192972b302ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:24.341562 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.341514 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e67e26-14bd-477e-b842-192972b302ab-kube-api-access-lc6mq" pod "e3e67e26-14bd-477e-b842-192972b302ab" (UID: "e3e67e26-14bd-477e-b842-192972b302ab"). InnerVolumeSpecName "kube-api-access-lc6mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:24.342461 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.342409 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e67e26-14bd-477e-b842-192972b302ab-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:24.342883 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.342586 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc6mq\" (UniqueName: \"kubernetes.io/projected/e3e67e26-14bd-477e-b842-192972b302ab-kube-api-access-lc6mq\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:24.342883 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.342619 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj4ls\" (UniqueName: \"kubernetes.io/projected/3405d103-6146-4e19-9055-106154b2c39c-kube-api-access-nj4ls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:24.448187 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.448081 5786 generic.go:356] "Generic (PLEG): container finished" podID="e3e67e26-14bd-477e-b842-192972b302ab" containerID="ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25" exitCode=0 Nov 04 01:30:24.448187 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.448159 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" event={"ID":"e3e67e26-14bd-477e-b842-192972b302ab","Type":"ContainerDied","Data":"ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25"} Nov 04 01:30:24.448426 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.448175 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" Nov 04 01:30:24.448426 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.448254 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d" event={"ID":"e3e67e26-14bd-477e-b842-192972b302ab","Type":"ContainerDied","Data":"5bec5bd585554d9793ccbc980e7e9a89fee07e03e1fb21025b10c71b696a5919"} Nov 04 01:30:24.448426 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.448285 5786 scope.go:122] "RemoveContainer" containerID="ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25" Nov 04 01:30:24.449608 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.449587 5786 generic.go:356] "Generic (PLEG): container finished" podID="3405d103-6146-4e19-9055-106154b2c39c" containerID="86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900" exitCode=0 Nov 04 01:30:24.449704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.449649 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" Nov 04 01:30:24.449704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.449672 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" event={"ID":"3405d103-6146-4e19-9055-106154b2c39c","Type":"ContainerDied","Data":"86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900"} Nov 04 01:30:24.449819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.449723 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj" event={"ID":"3405d103-6146-4e19-9055-106154b2c39c","Type":"ContainerDied","Data":"631a6dc6b2fd994a833d0d26a080df1920a2ab41e3ce44948b7bb306f7473932"} Nov 04 01:30:24.459649 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.459632 5786 scope.go:122] "RemoveContainer" containerID="ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25" Nov 04 01:30:24.459967 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:30:24.459938 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25\": container with ID starting with ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25 not found: ID does not exist" containerID="ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25" Nov 04 01:30:24.460058 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.459974 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25"} err="failed to get container status \"ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25\": rpc error: code = NotFound desc = could not find container \"ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25\": container with ID starting with ef65fa7f668efddddba0b729792137e64f3dfd595dfbac490acb1615dd616f25 not found: ID does not exist" Nov 04 01:30:24.460058 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.459995 5786 scope.go:122] "RemoveContainer" containerID="86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900" Nov 04 01:30:24.468840 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.468822 5786 scope.go:122] "RemoveContainer" containerID="86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900" Nov 04 01:30:24.469072 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:30:24.469053 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900\": container with ID starting with 86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900 not found: ID does not exist" containerID="86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900" Nov 04 01:30:24.469204 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.469078 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900"} err="failed to get container status \"86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900\": rpc error: code = NotFound desc = could not find container \"86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900\": container with ID starting with 86c3bc9dffd830dfee78a0ce1ef26c528930966ca38f95f126329410027dc900 not found: ID does not exist" Nov 04 01:30:24.726208 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.726127 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d"] Nov 04 01:30:24.731173 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.731146 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-9a0432-55bf44899-wn56d"] Nov 04 01:30:24.837551 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.837515 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj"] Nov 04 01:30:24.846377 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:24.846346 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-9a0432-8458445bb-4ghxj"] Nov 04 01:30:26.506185 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:26.506150 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3405d103-6146-4e19-9055-106154b2c39c" path="/var/lib/kubelet/pods/3405d103-6146-4e19-9055-106154b2c39c/volumes" Nov 04 01:30:26.506693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:26.506604 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3e67e26-14bd-477e-b842-192972b302ab" path="/var/lib/kubelet/pods/e3e67e26-14bd-477e-b842-192972b302ab/volumes" Nov 04 01:30:27.363496 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.363462 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv"] Nov 04 01:30:27.364183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364156 5786 cpu_manager.go:414] "containerMap: removing container" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" Nov 04 01:30:27.364183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364181 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364215 5786 cpu_manager.go:414] "containerMap: removing container" podUID="889517a5-9d9a-4643-9665-dd6b46496379" containerName="istio-proxy" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364224 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="889517a5-9d9a-4643-9665-dd6b46496379" containerName="istio-proxy" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364245 5786 cpu_manager.go:414] "containerMap: removing container" podUID="bb8d2819-e753-4b38-864b-49faf969feac" containerName="istio-proxy" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364254 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8d2819-e753-4b38-864b-49faf969feac" containerName="istio-proxy" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364261 5786 cpu_manager.go:414] "containerMap: removing container" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" containerName="cleanup" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364268 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" containerName="cleanup" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364279 5786 cpu_manager.go:414] "containerMap: removing container" podUID="e3e67e26-14bd-477e-b842-192972b302ab" containerName="payload-pre-processing" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364286 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3e67e26-14bd-477e-b842-192972b302ab" containerName="payload-pre-processing" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364308 5786 cpu_manager.go:414] "containerMap: removing container" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" containerName="cleanup" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364314 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" containerName="cleanup" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364348 5786 cpu_manager.go:414] "containerMap: removing container" podUID="3405d103-6146-4e19-9055-106154b2c39c" containerName="payload-processing" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364356 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3405d103-6146-4e19-9055-106154b2c39c" containerName="payload-processing" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364374 5786 cpu_manager.go:414] "containerMap: removing container" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" containerName="cleanup" Nov 04 01:30:27.364398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364381 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" containerName="cleanup" Nov 04 01:30:27.365133 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364526 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="3405d103-6146-4e19-9055-106154b2c39c" containerName="payload-processing" Nov 04 01:30:27.365133 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364539 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" containerName="cleanup" Nov 04 01:30:27.365133 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364549 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b454222-560a-4d7d-8b07-997a983463c7" containerName="istio-proxy" Nov 04 01:30:27.365133 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364565 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="889517a5-9d9a-4643-9665-dd6b46496379" containerName="istio-proxy" Nov 04 01:30:27.365133 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364577 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8d2819-e753-4b38-864b-49faf969feac" containerName="istio-proxy" Nov 04 01:30:27.365133 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364586 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" containerName="cleanup" Nov 04 01:30:27.365133 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364596 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3e67e26-14bd-477e-b842-192972b302ab" containerName="payload-pre-processing" Nov 04 01:30:27.365133 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.364606 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb562218-5de4-425e-9226-2c3fe590fda9" containerName="cleanup" Nov 04 01:30:27.369299 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.369278 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" Nov 04 01:30:27.466142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.465592 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv"] Nov 04 01:30:27.525145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.501456 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-fc46df\"" Nov 04 01:30:27.525145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.502513 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-fc46df-dockercfg-bd7kc\"" Nov 04 01:30:27.578141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.556300 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbbsj\" (UniqueName: \"kubernetes.io/projected/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-kube-api-access-fbbsj\") pod \"payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv\" (UID: \"e496528c-9914-4f1f-bb29-f9d10ac5b3b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" Nov 04 01:30:27.578141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.556380 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv\" (UID: \"e496528c-9914-4f1f-bb29-f9d10ac5b3b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" Nov 04 01:30:27.578141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.575594 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d"] Nov 04 01:30:27.588404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.588377 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" Nov 04 01:30:27.611495 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.611458 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d"] Nov 04 01:30:27.658112 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.658032 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbbsj\" (UniqueName: \"kubernetes.io/projected/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-kube-api-access-fbbsj\") pod \"payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv\" (UID: \"e496528c-9914-4f1f-bb29-f9d10ac5b3b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" Nov 04 01:30:27.658112 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.658084 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv\" (UID: \"e496528c-9914-4f1f-bb29-f9d10ac5b3b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" Nov 04 01:30:27.658859 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.658839 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv\" (UID: \"e496528c-9914-4f1f-bb29-f9d10ac5b3b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" Nov 04 01:30:27.715975 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.715944 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbbsj\" (UniqueName: \"kubernetes.io/projected/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-kube-api-access-fbbsj\") pod \"payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv\" (UID: \"e496528c-9914-4f1f-bb29-f9d10ac5b3b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" Nov 04 01:30:27.759718 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.759686 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnkwn\" (UniqueName: \"kubernetes.io/projected/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-kube-api-access-mnkwn\") pod \"payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d\" (UID: \"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" Nov 04 01:30:27.759878 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.759735 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d\" (UID: \"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" Nov 04 01:30:27.861267 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.861234 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnkwn\" (UniqueName: \"kubernetes.io/projected/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-kube-api-access-mnkwn\") pod \"payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d\" (UID: \"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" Nov 04 01:30:27.861444 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.861283 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d\" (UID: \"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" Nov 04 01:30:27.861986 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.861965 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d\" (UID: \"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" Nov 04 01:30:27.919065 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.918997 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnkwn\" (UniqueName: \"kubernetes.io/projected/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-kube-api-access-mnkwn\") pod \"payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d\" (UID: \"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" Nov 04 01:30:27.983558 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:27.983514 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" Nov 04 01:30:28.204090 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:28.204009 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" Nov 04 01:30:28.362398 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:28.362365 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv"] Nov 04 01:30:28.367380 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:30:28.367354 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode496528c_9914_4f1f_bb29_f9d10ac5b3b2.slice/crio-52d3120bcf8fca615dde9cc31fbe3c3bb4f16a0b70662171ab2c0645610e0a02 WatchSource:0}: Error finding container 52d3120bcf8fca615dde9cc31fbe3c3bb4f16a0b70662171ab2c0645610e0a02: Status 404 returned error can't find the container with id 52d3120bcf8fca615dde9cc31fbe3c3bb4f16a0b70662171ab2c0645610e0a02 Nov 04 01:30:28.546876 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:28.546782 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" event={"ID":"e496528c-9914-4f1f-bb29-f9d10ac5b3b2","Type":"ContainerStarted","Data":"3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1"} Nov 04 01:30:28.546876 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:28.546843 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" event={"ID":"e496528c-9914-4f1f-bb29-f9d10ac5b3b2","Type":"ContainerStarted","Data":"52d3120bcf8fca615dde9cc31fbe3c3bb4f16a0b70662171ab2c0645610e0a02"} Nov 04 01:30:28.547320 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:28.546970 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" Nov 04 01:30:28.579960 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:28.579867 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d"] Nov 04 01:30:28.584371 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:30:28.584343 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978c1ab7_fbe9_4cfd_a7c6_d5d8f5ca64ac.slice/crio-bb14843f8c1efad9f1a0dfd99f0bf38a542108caff5c3c5c9b3fa76c19fd576a WatchSource:0}: Error finding container bb14843f8c1efad9f1a0dfd99f0bf38a542108caff5c3c5c9b3fa76c19fd576a: Status 404 returned error can't find the container with id bb14843f8c1efad9f1a0dfd99f0bf38a542108caff5c3c5c9b3fa76c19fd576a Nov 04 01:30:28.662969 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:28.662921 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" podStartSLOduration=1.662908002 podStartE2EDuration="1.662908002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:28.661820429 +0000 UTC m=+1974.278166194" watchObservedRunningTime="2026-11-04 01:30:28.662908002 +0000 UTC m=+1974.279253764" Nov 04 01:30:29.554970 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:29.554932 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" event={"ID":"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac","Type":"ContainerStarted","Data":"8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1"} Nov 04 01:30:29.555419 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:29.554994 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" event={"ID":"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac","Type":"ContainerStarted","Data":"bb14843f8c1efad9f1a0dfd99f0bf38a542108caff5c3c5c9b3fa76c19fd576a"} Nov 04 01:30:29.555419 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:29.555233 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" Nov 04 01:30:29.673902 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:29.673848 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" podStartSLOduration=2.673834062 podStartE2EDuration="2.673834062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:29.6736131 +0000 UTC m=+1975.289958864" watchObservedRunningTime="2026-11-04 01:30:29.673834062 +0000 UTC m=+1975.290179826" Nov 04 01:30:39.558280 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:39.558250 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" Nov 04 01:30:40.564992 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:40.564964 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" Nov 04 01:30:45.087946 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:30:45.087860 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:30:47.833684 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:47.833648 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v"] Nov 04 01:30:52.834569 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:52.834500 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" podUID="38a241a8-c4a4-4a4e-9dbc-fbe047089b0c" containerName="payload-processing" containerID="cri-o://281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74" gracePeriod=25 Nov 04 01:30:53.125623 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.125598 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" Nov 04 01:30:53.200000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.199964 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-plugins-config-volume\") pod \"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c\" (UID: \"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c\") " Nov 04 01:30:53.200192 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.200066 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-kube-api-access-m6t9v\" (UniqueName: \"kubernetes.io/projected/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-kube-api-access-m6t9v\") pod \"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c\" (UID: \"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c\") " Nov 04 01:30:53.200394 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.200370 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-plugins-config-volume" pod "38a241a8-c4a4-4a4e-9dbc-fbe047089b0c" (UID: "38a241a8-c4a4-4a4e-9dbc-fbe047089b0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:53.200555 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.200542 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:53.202436 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.202395 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-kube-api-access-m6t9v" pod "38a241a8-c4a4-4a4e-9dbc-fbe047089b0c" (UID: "38a241a8-c4a4-4a4e-9dbc-fbe047089b0c"). InnerVolumeSpecName "kube-api-access-m6t9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:53.301685 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.301644 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6t9v\" (UniqueName: \"kubernetes.io/projected/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c-kube-api-access-m6t9v\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:53.750466 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.750427 5786 generic.go:356] "Generic (PLEG): container finished" podID="38a241a8-c4a4-4a4e-9dbc-fbe047089b0c" containerID="281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74" exitCode=0 Nov 04 01:30:53.750648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.750469 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" Nov 04 01:30:53.750648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.750511 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" event={"ID":"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c","Type":"ContainerDied","Data":"281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74"} Nov 04 01:30:53.750648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.750570 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v" event={"ID":"38a241a8-c4a4-4a4e-9dbc-fbe047089b0c","Type":"ContainerDied","Data":"09b2a40bbfc1ce0c79d7e4f08d667fba1ab5247d60e0a2329b63d4a60c3ac0b1"} Nov 04 01:30:53.750648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.750600 5786 scope.go:122] "RemoveContainer" containerID="281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74" Nov 04 01:30:53.761079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.761062 5786 scope.go:122] "RemoveContainer" containerID="281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74" Nov 04 01:30:53.761400 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:30:53.761378 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74\": container with ID starting with 281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74 not found: ID does not exist" containerID="281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74" Nov 04 01:30:53.761492 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.761409 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74"} err="failed to get container status \"281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74\": rpc error: code = NotFound desc = could not find container \"281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74\": container with ID starting with 281357df757c7779ea4f6707349fd6e0a2d7259c7e8b35bd6e1a682747ab7f74 not found: ID does not exist" Nov 04 01:30:53.916646 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.916600 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v"] Nov 04 01:30:53.964054 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:53.964021 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5186fe-bf54d4cd-9rn9v"] Nov 04 01:30:54.506735 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:30:54.506702 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a241a8-c4a4-4a4e-9dbc-fbe047089b0c" path="/var/lib/kubelet/pods/38a241a8-c4a4-4a4e-9dbc-fbe047089b0c/volumes" Nov 04 01:31:05.014934 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:05.014897 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:31:12.236635 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:12.236600 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c"] Nov 04 01:31:12.351019 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:12.350988 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc"] Nov 04 01:31:15.087905 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:31:15.087866 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:31:17.237404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.237363 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" podUID="a5862d90-590b-4eb4-bd40-8ca937bccf4c" containerName="payload-processing" containerID="cri-o://3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae" gracePeriod=25 Nov 04 01:31:17.351639 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.351595 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" podUID="f87c612a-502f-4488-b9d1-57f1d4bee521" containerName="payload-pre-processing" containerID="cri-o://3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff" gracePeriod=25 Nov 04 01:31:17.519656 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.519630 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" Nov 04 01:31:17.629340 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.629315 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" Nov 04 01:31:17.662482 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.662450 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5862d90-590b-4eb4-bd40-8ca937bccf4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5862d90-590b-4eb4-bd40-8ca937bccf4c-plugins-config-volume\") pod \"a5862d90-590b-4eb4-bd40-8ca937bccf4c\" (UID: \"a5862d90-590b-4eb4-bd40-8ca937bccf4c\") " Nov 04 01:31:17.662680 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.662596 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5862d90-590b-4eb4-bd40-8ca937bccf4c-kube-api-access-ttg2r\" (UniqueName: \"kubernetes.io/projected/a5862d90-590b-4eb4-bd40-8ca937bccf4c-kube-api-access-ttg2r\") pod \"a5862d90-590b-4eb4-bd40-8ca937bccf4c\" (UID: \"a5862d90-590b-4eb4-bd40-8ca937bccf4c\") " Nov 04 01:31:17.662837 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.662813 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5862d90-590b-4eb4-bd40-8ca937bccf4c-plugins-config-volume" pod "a5862d90-590b-4eb4-bd40-8ca937bccf4c" (UID: "a5862d90-590b-4eb4-bd40-8ca937bccf4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:17.663404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.663376 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5862d90-590b-4eb4-bd40-8ca937bccf4c-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:17.664905 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.664877 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5862d90-590b-4eb4-bd40-8ca937bccf4c-kube-api-access-ttg2r" pod "a5862d90-590b-4eb4-bd40-8ca937bccf4c" (UID: "a5862d90-590b-4eb4-bd40-8ca937bccf4c"). InnerVolumeSpecName "kube-api-access-ttg2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:17.764275 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.764186 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f87c612a-502f-4488-b9d1-57f1d4bee521-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87c612a-502f-4488-b9d1-57f1d4bee521-plugins-config-volume\") pod \"f87c612a-502f-4488-b9d1-57f1d4bee521\" (UID: \"f87c612a-502f-4488-b9d1-57f1d4bee521\") " Nov 04 01:31:17.764509 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.764492 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f87c612a-502f-4488-b9d1-57f1d4bee521-kube-api-access-482lf\" (UniqueName: \"kubernetes.io/projected/f87c612a-502f-4488-b9d1-57f1d4bee521-kube-api-access-482lf\") pod \"f87c612a-502f-4488-b9d1-57f1d4bee521\" (UID: \"f87c612a-502f-4488-b9d1-57f1d4bee521\") " Nov 04 01:31:17.764567 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.764524 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87c612a-502f-4488-b9d1-57f1d4bee521-plugins-config-volume" pod "f87c612a-502f-4488-b9d1-57f1d4bee521" (UID: "f87c612a-502f-4488-b9d1-57f1d4bee521"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:17.764979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.764958 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttg2r\" (UniqueName: \"kubernetes.io/projected/a5862d90-590b-4eb4-bd40-8ca937bccf4c-kube-api-access-ttg2r\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:17.765081 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.764985 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87c612a-502f-4488-b9d1-57f1d4bee521-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:17.766642 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.766621 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87c612a-502f-4488-b9d1-57f1d4bee521-kube-api-access-482lf" pod "f87c612a-502f-4488-b9d1-57f1d4bee521" (UID: "f87c612a-502f-4488-b9d1-57f1d4bee521"). InnerVolumeSpecName "kube-api-access-482lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:17.866292 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:17.866254 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-482lf\" (UniqueName: \"kubernetes.io/projected/f87c612a-502f-4488-b9d1-57f1d4bee521-kube-api-access-482lf\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:18.140728 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.140693 5786 generic.go:356] "Generic (PLEG): container finished" podID="a5862d90-590b-4eb4-bd40-8ca937bccf4c" containerID="3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae" exitCode=0 Nov 04 01:31:18.140904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.140744 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" Nov 04 01:31:18.140904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.140779 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" event={"ID":"a5862d90-590b-4eb4-bd40-8ca937bccf4c","Type":"ContainerDied","Data":"3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae"} Nov 04 01:31:18.140904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.140834 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c" event={"ID":"a5862d90-590b-4eb4-bd40-8ca937bccf4c","Type":"ContainerDied","Data":"1d0307ca13e96fb3723ab89857809677b016d13fb77b3c62964fb974e4fff9bc"} Nov 04 01:31:18.140904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.140858 5786 scope.go:122] "RemoveContainer" containerID="3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae" Nov 04 01:31:18.142204 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.142184 5786 generic.go:356] "Generic (PLEG): container finished" podID="f87c612a-502f-4488-b9d1-57f1d4bee521" containerID="3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff" exitCode=0 Nov 04 01:31:18.142283 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.142230 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" Nov 04 01:31:18.142328 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.142228 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" event={"ID":"f87c612a-502f-4488-b9d1-57f1d4bee521","Type":"ContainerDied","Data":"3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff"} Nov 04 01:31:18.142371 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.142348 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc" event={"ID":"f87c612a-502f-4488-b9d1-57f1d4bee521","Type":"ContainerDied","Data":"8aa3bf50eee2fce793455d27a9e3dfd3c9c29b58ce4e7c11e43af7d2151783ba"} Nov 04 01:31:18.150945 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.150926 5786 scope.go:122] "RemoveContainer" containerID="3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae" Nov 04 01:31:18.151372 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:31:18.151347 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae\": container with ID starting with 3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae not found: ID does not exist" containerID="3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae" Nov 04 01:31:18.151484 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.151379 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae"} err="failed to get container status \"3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae\": rpc error: code = NotFound desc = could not find container \"3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae\": container with ID starting with 3ef95c2ac0f12006879b19e7912dba59e5d7e1dec84d70cff265d9a3eb4451ae not found: ID does not exist" Nov 04 01:31:18.151484 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.151396 5786 scope.go:122] "RemoveContainer" containerID="3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff" Nov 04 01:31:18.162227 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.162202 5786 scope.go:122] "RemoveContainer" containerID="3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff" Nov 04 01:31:18.162527 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:31:18.162505 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff\": container with ID starting with 3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff not found: ID does not exist" containerID="3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff" Nov 04 01:31:18.162577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.162534 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff"} err="failed to get container status \"3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff\": rpc error: code = NotFound desc = could not find container \"3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff\": container with ID starting with 3de7a566401816a97ef5649c988394cdab157c5a249b31c09a19e1ac4a0f3fff not found: ID does not exist" Nov 04 01:31:18.307913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.307883 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c"] Nov 04 01:31:18.315163 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.315140 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3dc6c8-85d44b6db4-96n5c"] Nov 04 01:31:18.462929 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.462835 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc"] Nov 04 01:31:18.467974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.467947 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3dc6c8-7d5b48b48c-l5thc"] Nov 04 01:31:18.506480 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.506448 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5862d90-590b-4eb4-bd40-8ca937bccf4c" path="/var/lib/kubelet/pods/a5862d90-590b-4eb4-bd40-8ca937bccf4c/volumes" Nov 04 01:31:18.506870 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:18.506857 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f87c612a-502f-4488-b9d1-57f1d4bee521" path="/var/lib/kubelet/pods/f87c612a-502f-4488-b9d1-57f1d4bee521/volumes" Nov 04 01:31:20.514320 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.514288 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57"] Nov 04 01:31:20.514905 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.514885 5786 cpu_manager.go:414] "containerMap: removing container" podUID="38a241a8-c4a4-4a4e-9dbc-fbe047089b0c" containerName="payload-processing" Nov 04 01:31:20.514969 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.514907 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a241a8-c4a4-4a4e-9dbc-fbe047089b0c" containerName="payload-processing" Nov 04 01:31:20.514969 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.514922 5786 cpu_manager.go:414] "containerMap: removing container" podUID="f87c612a-502f-4488-b9d1-57f1d4bee521" containerName="payload-pre-processing" Nov 04 01:31:20.514969 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.514927 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f87c612a-502f-4488-b9d1-57f1d4bee521" containerName="payload-pre-processing" Nov 04 01:31:20.514969 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.514938 5786 cpu_manager.go:414] "containerMap: removing container" podUID="a5862d90-590b-4eb4-bd40-8ca937bccf4c" containerName="payload-processing" Nov 04 01:31:20.514969 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.514943 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5862d90-590b-4eb4-bd40-8ca937bccf4c" containerName="payload-processing" Nov 04 01:31:20.515145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.515062 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a241a8-c4a4-4a4e-9dbc-fbe047089b0c" containerName="payload-processing" Nov 04 01:31:20.515145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.515071 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="f87c612a-502f-4488-b9d1-57f1d4bee521" containerName="payload-pre-processing" Nov 04 01:31:20.515145 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.515077 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5862d90-590b-4eb4-bd40-8ca937bccf4c" containerName="payload-processing" Nov 04 01:31:20.519874 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.519851 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:20.571581 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.571551 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-fc46df-serving-cert\"" Nov 04 01:31:20.621694 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.621660 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57"] Nov 04 01:31:20.703768 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.703720 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:20.703951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.703864 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtdj\" (UniqueName: \"kubernetes.io/projected/ce0ed274-cc86-4ab4-b8fb-db31ee650267-kube-api-access-tvtdj\") pod \"maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:20.703951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.703942 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0ed274-cc86-4ab4-b8fb-db31ee650267-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:20.704047 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.704023 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:20.807205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.806592 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0ed274-cc86-4ab4-b8fb-db31ee650267-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:20.807205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.806681 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:20.807205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.806757 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:20.807205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.806894 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtdj\" (UniqueName: \"kubernetes.io/projected/ce0ed274-cc86-4ab4-b8fb-db31ee650267-kube-api-access-tvtdj\") pod \"maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:20.829525 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.821514 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0ed274-cc86-4ab4-b8fb-db31ee650267-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:20.829525 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.828103 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:20.835320 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.835290 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:20.864577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:20.864539 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtdj\" (UniqueName: \"kubernetes.io/projected/ce0ed274-cc86-4ab4-b8fb-db31ee650267-kube-api-access-tvtdj\") pod \"maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:21.137350 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:21.137314 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:21.515833 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:21.515681 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57"] Nov 04 01:31:21.520838 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:31:21.520811 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0ed274_cc86_4ab4_b8fb_db31ee650267.slice/crio-a09ff1b839c45f09ab6e7c3fc673179d39d59722607c160da65aede839e71f53 WatchSource:0}: Error finding container a09ff1b839c45f09ab6e7c3fc673179d39d59722607c160da65aede839e71f53: Status 404 returned error can't find the container with id a09ff1b839c45f09ab6e7c3fc673179d39d59722607c160da65aede839e71f53 Nov 04 01:31:22.172104 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:22.172065 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" event={"ID":"ce0ed274-cc86-4ab4-b8fb-db31ee650267","Type":"ContainerStarted","Data":"a09ff1b839c45f09ab6e7c3fc673179d39d59722607c160da65aede839e71f53"} Nov 04 01:31:23.179494 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:23.179461 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" event={"ID":"ce0ed274-cc86-4ab4-b8fb-db31ee650267","Type":"ContainerStarted","Data":"135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10"} Nov 04 01:31:23.179946 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:23.179597 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:23.297600 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:23.297547 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" podStartSLOduration=2.411503271 podStartE2EDuration="3.297533915s" totalImagesPullingTime="886.030644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:23.295693068 +0000 UTC m=+2028.912038831" watchObservedRunningTime="2026-11-04 01:31:23.297533915 +0000 UTC m=+2028.913879678" Nov 04 01:31:27.733850 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:27.733810 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7"] Nov 04 01:31:27.867688 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:27.867654 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm"] Nov 04 01:31:29.191669 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:29.191635 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:31:31.811059 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:31.811024 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:31:32.734142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:32.734088 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" podUID="97d6013a-d49b-4639-88dd-f5e458f50d28" containerName="payload-processing" containerID="cri-o://05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b" gracePeriod=25 Nov 04 01:31:32.868808 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:32.868770 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" podUID="d198877f-a0af-4c3a-8902-160811220bd8" containerName="payload-pre-processing" containerID="cri-o://c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f" gracePeriod=25 Nov 04 01:31:33.006216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.006188 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" Nov 04 01:31:33.032396 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.032363 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d6013a-d49b-4639-88dd-f5e458f50d28-kube-api-access-lmng9\" (UniqueName: \"kubernetes.io/projected/97d6013a-d49b-4639-88dd-f5e458f50d28-kube-api-access-lmng9\") pod \"97d6013a-d49b-4639-88dd-f5e458f50d28\" (UID: \"97d6013a-d49b-4639-88dd-f5e458f50d28\") " Nov 04 01:31:33.032549 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.032444 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d6013a-d49b-4639-88dd-f5e458f50d28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d6013a-d49b-4639-88dd-f5e458f50d28-plugins-config-volume\") pod \"97d6013a-d49b-4639-88dd-f5e458f50d28\" (UID: \"97d6013a-d49b-4639-88dd-f5e458f50d28\") " Nov 04 01:31:33.032818 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.032795 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d6013a-d49b-4639-88dd-f5e458f50d28-plugins-config-volume" pod "97d6013a-d49b-4639-88dd-f5e458f50d28" (UID: "97d6013a-d49b-4639-88dd-f5e458f50d28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:33.032996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.032982 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d6013a-d49b-4639-88dd-f5e458f50d28-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:33.034533 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.034513 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d6013a-d49b-4639-88dd-f5e458f50d28-kube-api-access-lmng9" pod "97d6013a-d49b-4639-88dd-f5e458f50d28" (UID: "97d6013a-d49b-4639-88dd-f5e458f50d28"). InnerVolumeSpecName "kube-api-access-lmng9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:33.122624 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.122594 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" Nov 04 01:31:33.134188 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.134161 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d198877f-a0af-4c3a-8902-160811220bd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d198877f-a0af-4c3a-8902-160811220bd8-plugins-config-volume\") pod \"d198877f-a0af-4c3a-8902-160811220bd8\" (UID: \"d198877f-a0af-4c3a-8902-160811220bd8\") " Nov 04 01:31:33.134358 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.134343 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d198877f-a0af-4c3a-8902-160811220bd8-kube-api-access-vhh5h\" (UniqueName: \"kubernetes.io/projected/d198877f-a0af-4c3a-8902-160811220bd8-kube-api-access-vhh5h\") pod \"d198877f-a0af-4c3a-8902-160811220bd8\" (UID: \"d198877f-a0af-4c3a-8902-160811220bd8\") " Nov 04 01:31:33.134615 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.134587 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d198877f-a0af-4c3a-8902-160811220bd8-plugins-config-volume" pod "d198877f-a0af-4c3a-8902-160811220bd8" (UID: "d198877f-a0af-4c3a-8902-160811220bd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:33.134708 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.134647 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmng9\" (UniqueName: \"kubernetes.io/projected/97d6013a-d49b-4639-88dd-f5e458f50d28-kube-api-access-lmng9\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:33.136431 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.136400 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d198877f-a0af-4c3a-8902-160811220bd8-kube-api-access-vhh5h" pod "d198877f-a0af-4c3a-8902-160811220bd8" (UID: "d198877f-a0af-4c3a-8902-160811220bd8"). InnerVolumeSpecName "kube-api-access-vhh5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:33.236388 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.236314 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d198877f-a0af-4c3a-8902-160811220bd8-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:33.236388 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.236338 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhh5h\" (UniqueName: \"kubernetes.io/projected/d198877f-a0af-4c3a-8902-160811220bd8-kube-api-access-vhh5h\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:33.292182 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.292151 5786 generic.go:356] "Generic (PLEG): container finished" podID="d198877f-a0af-4c3a-8902-160811220bd8" containerID="c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f" exitCode=0 Nov 04 01:31:33.292365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.292198 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" Nov 04 01:31:33.292365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.292235 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" event={"ID":"d198877f-a0af-4c3a-8902-160811220bd8","Type":"ContainerDied","Data":"c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f"} Nov 04 01:31:33.292365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.292291 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm" event={"ID":"d198877f-a0af-4c3a-8902-160811220bd8","Type":"ContainerDied","Data":"17e0ab8076e94b49c25afccb0a63ce03ad22a226d95e727e5a598a4f5b89238b"} Nov 04 01:31:33.292365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.292320 5786 scope.go:122] "RemoveContainer" containerID="c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f" Nov 04 01:31:33.293730 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.293708 5786 generic.go:356] "Generic (PLEG): container finished" podID="97d6013a-d49b-4639-88dd-f5e458f50d28" containerID="05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b" exitCode=0 Nov 04 01:31:33.293847 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.293763 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" Nov 04 01:31:33.293847 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.293802 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" event={"ID":"97d6013a-d49b-4639-88dd-f5e458f50d28","Type":"ContainerDied","Data":"05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b"} Nov 04 01:31:33.293941 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.293853 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7" event={"ID":"97d6013a-d49b-4639-88dd-f5e458f50d28","Type":"ContainerDied","Data":"598556e0bfe41f3262c973e29fdc69c0c0ba1818fdc9b8ea102b40d0553b8e37"} Nov 04 01:31:33.309446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.303953 5786 scope.go:122] "RemoveContainer" containerID="c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f" Nov 04 01:31:33.309446 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:31:33.304340 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f\": container with ID starting with c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f not found: ID does not exist" containerID="c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f" Nov 04 01:31:33.309446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.304375 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f"} err="failed to get container status \"c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f\": rpc error: code = NotFound desc = could not find container \"c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f\": container with ID starting with c403ac0653a68466cad4a311d56009267b9f3d57d4923b5c3275206c7bf4963f not found: ID does not exist" Nov 04 01:31:33.309446 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.304413 5786 scope.go:122] "RemoveContainer" containerID="05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b" Nov 04 01:31:33.317991 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.317970 5786 scope.go:122] "RemoveContainer" containerID="05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b" Nov 04 01:31:33.318315 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:31:33.318296 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b\": container with ID starting with 05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b not found: ID does not exist" containerID="05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b" Nov 04 01:31:33.318390 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.318321 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b"} err="failed to get container status \"05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b\": rpc error: code = NotFound desc = could not find container \"05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b\": container with ID starting with 05f7d9a6204aaf24cd4f749769b29d3165d8b7e1c1134df8ffabea2f61699a0b not found: ID does not exist" Nov 04 01:31:33.452044 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.451723 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm"] Nov 04 01:31:33.455549 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.455525 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-42958c-54fd98d7c5-t4jbm"] Nov 04 01:31:33.560238 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.560137 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7"] Nov 04 01:31:33.562714 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:33.562691 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-42958c-d5965db48-v6lc7"] Nov 04 01:31:34.506078 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:34.506039 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d6013a-d49b-4639-88dd-f5e458f50d28" path="/var/lib/kubelet/pods/97d6013a-d49b-4639-88dd-f5e458f50d28/volumes" Nov 04 01:31:34.506654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:34.506633 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d198877f-a0af-4c3a-8902-160811220bd8" path="/var/lib/kubelet/pods/d198877f-a0af-4c3a-8902-160811220bd8/volumes" Nov 04 01:31:45.088865 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:31:45.088826 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:31:45.425254 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.425223 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k"] Nov 04 01:31:45.425438 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.425418 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" podUID="f5182da9-f486-453f-80b3-2946aae607fa" containerName="maas-api" containerID="cri-o://5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb" gracePeriod=30 Nov 04 01:31:45.545844 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.545810 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" podUID="f5182da9-f486-453f-80b3-2946aae607fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.112:8443/health\": dial tcp 10.132.0.112:8443: connect: connection refused" Nov 04 01:31:45.781324 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.781292 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:31:45.850164 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.836722 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-maas-api-tls\") pod \"f5182da9-f486-453f-80b3-2946aae607fa\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " Nov 04 01:31:45.850164 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.836974 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5182da9-f486-453f-80b3-2946aae607fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5182da9-f486-453f-80b3-2946aae607fa-openshift-service-ca\") pod \"f5182da9-f486-453f-80b3-2946aae607fa\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " Nov 04 01:31:45.850164 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.837160 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5182da9-f486-453f-80b3-2946aae607fa-kube-api-access-vwhlz\" (UniqueName: \"kubernetes.io/projected/f5182da9-f486-453f-80b3-2946aae607fa-kube-api-access-vwhlz\") pod \"f5182da9-f486-453f-80b3-2946aae607fa\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " Nov 04 01:31:45.850164 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.837240 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-metrics-certs\") pod \"f5182da9-f486-453f-80b3-2946aae607fa\" (UID: \"f5182da9-f486-453f-80b3-2946aae607fa\") " Nov 04 01:31:45.850164 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.840829 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5182da9-f486-453f-80b3-2946aae607fa-openshift-service-ca" pod "f5182da9-f486-453f-80b3-2946aae607fa" (UID: "f5182da9-f486-453f-80b3-2946aae607fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:45.859857 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.859782 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-maas-api-tls" pod "f5182da9-f486-453f-80b3-2946aae607fa" (UID: "f5182da9-f486-453f-80b3-2946aae607fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:31:45.882549 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.882482 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-metrics-certs" pod "f5182da9-f486-453f-80b3-2946aae607fa" (UID: "f5182da9-f486-453f-80b3-2946aae607fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:31:45.883679 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.883636 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5182da9-f486-453f-80b3-2946aae607fa-kube-api-access-vwhlz" pod "f5182da9-f486-453f-80b3-2946aae607fa" (UID: "f5182da9-f486-453f-80b3-2946aae607fa"). InnerVolumeSpecName "kube-api-access-vwhlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:45.939047 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.939009 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5182da9-f486-453f-80b3-2946aae607fa-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:45.939047 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.939043 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwhlz\" (UniqueName: \"kubernetes.io/projected/f5182da9-f486-453f-80b3-2946aae607fa-kube-api-access-vwhlz\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:45.939284 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.939061 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:45.939284 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:45.939074 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5182da9-f486-453f-80b3-2946aae607fa-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:46.376962 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:46.376930 5786 generic.go:356] "Generic (PLEG): container finished" podID="f5182da9-f486-453f-80b3-2946aae607fa" containerID="5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb" exitCode=0 Nov 04 01:31:46.377525 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:46.376974 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" Nov 04 01:31:46.377525 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:46.377018 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" event={"ID":"f5182da9-f486-453f-80b3-2946aae607fa","Type":"ContainerDied","Data":"5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb"} Nov 04 01:31:46.377525 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:46.377080 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k" event={"ID":"f5182da9-f486-453f-80b3-2946aae607fa","Type":"ContainerDied","Data":"90c48b656a09e882f89959d5348b7a4fe484afe10a3eae00df7dea77f871550e"} Nov 04 01:31:46.377525 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:46.377115 5786 scope.go:122] "RemoveContainer" containerID="5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb" Nov 04 01:31:46.389052 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:46.389032 5786 scope.go:122] "RemoveContainer" containerID="5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb" Nov 04 01:31:46.389452 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:31:46.389417 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb\": container with ID starting with 5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb not found: ID does not exist" containerID="5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb" Nov 04 01:31:46.389574 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:46.389462 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb"} err="failed to get container status \"5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb\": rpc error: code = NotFound desc = could not find container \"5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb\": container with ID starting with 5d14a86278a704d03aeecde0810e70919eb6a2db7808620968da8ba1f0ac85bb not found: ID does not exist" Nov 04 01:31:46.560952 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:46.560916 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k"] Nov 04 01:31:46.563768 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:46.563743 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-3dc6c8-655f9d4d4b-jxb9k"] Nov 04 01:31:48.505469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:48.505438 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5182da9-f486-453f-80b3-2946aae607fa" path="/var/lib/kubelet/pods/f5182da9-f486-453f-80b3-2946aae607fa/volumes" Nov 04 01:31:58.662206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:31:58.662156 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:32:01.157184 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:01.157142 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt"] Nov 04 01:32:13.625795 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:13.625757 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9"] Nov 04 01:32:13.626786 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:13.626751 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" podUID="cf329b96-b5f3-4c89-be83-0e107880e64f" containerName="maas-api" containerID="cri-o://9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62" gracePeriod=30 Nov 04 01:32:13.978352 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:13.978319 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:32:14.047204 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.046465 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf329b96-b5f3-4c89-be83-0e107880e64f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf329b96-b5f3-4c89-be83-0e107880e64f-openshift-service-ca\") pod \"cf329b96-b5f3-4c89-be83-0e107880e64f\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " Nov 04 01:32:14.047204 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.046646 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf329b96-b5f3-4c89-be83-0e107880e64f-kube-api-access-kbpq8\" (UniqueName: \"kubernetes.io/projected/cf329b96-b5f3-4c89-be83-0e107880e64f-kube-api-access-kbpq8\") pod \"cf329b96-b5f3-4c89-be83-0e107880e64f\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " Nov 04 01:32:14.047204 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.046781 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-metrics-certs\") pod \"cf329b96-b5f3-4c89-be83-0e107880e64f\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " Nov 04 01:32:14.047204 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.046847 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-maas-api-tls\") pod \"cf329b96-b5f3-4c89-be83-0e107880e64f\" (UID: \"cf329b96-b5f3-4c89-be83-0e107880e64f\") " Nov 04 01:32:14.060227 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.056243 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf329b96-b5f3-4c89-be83-0e107880e64f-openshift-service-ca" pod "cf329b96-b5f3-4c89-be83-0e107880e64f" (UID: "cf329b96-b5f3-4c89-be83-0e107880e64f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:14.088269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.079207 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-maas-api-tls" pod "cf329b96-b5f3-4c89-be83-0e107880e64f" (UID: "cf329b96-b5f3-4c89-be83-0e107880e64f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:14.088269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.081366 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-metrics-certs" pod "cf329b96-b5f3-4c89-be83-0e107880e64f" (UID: "cf329b96-b5f3-4c89-be83-0e107880e64f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:14.098559 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.098520 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf329b96-b5f3-4c89-be83-0e107880e64f-kube-api-access-kbpq8" pod "cf329b96-b5f3-4c89-be83-0e107880e64f" (UID: "cf329b96-b5f3-4c89-be83-0e107880e64f"). InnerVolumeSpecName "kube-api-access-kbpq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:14.148674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.148628 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf329b96-b5f3-4c89-be83-0e107880e64f-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:14.148674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.148670 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbpq8\" (UniqueName: \"kubernetes.io/projected/cf329b96-b5f3-4c89-be83-0e107880e64f-kube-api-access-kbpq8\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:14.148882 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.148689 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:14.148882 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.148703 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf329b96-b5f3-4c89-be83-0e107880e64f-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:14.575789 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.575749 5786 generic.go:356] "Generic (PLEG): container finished" podID="cf329b96-b5f3-4c89-be83-0e107880e64f" containerID="9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62" exitCode=0 Nov 04 01:32:14.575956 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.575797 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" Nov 04 01:32:14.575956 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.575834 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" event={"ID":"cf329b96-b5f3-4c89-be83-0e107880e64f","Type":"ContainerDied","Data":"9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62"} Nov 04 01:32:14.575956 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.575892 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9" event={"ID":"cf329b96-b5f3-4c89-be83-0e107880e64f","Type":"ContainerDied","Data":"d32d75a4ae77dab96176a4ea4e0fbf6def494ea5b853d4f241cc816511682cbe"} Nov 04 01:32:14.575956 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.575921 5786 scope.go:122] "RemoveContainer" containerID="9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62" Nov 04 01:32:14.617143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.609538 5786 scope.go:122] "RemoveContainer" containerID="9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62" Nov 04 01:32:14.617143 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:14.610551 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62\": container with ID starting with 9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62 not found: ID does not exist" containerID="9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62" Nov 04 01:32:14.617143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.610591 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62"} err="failed to get container status \"9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62\": rpc error: code = NotFound desc = could not find container \"9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62\": container with ID starting with 9f8b707a71f2f7d41c7e975a76812affe03fd07d3a2c7cca0360b62e05175d62 not found: ID does not exist" Nov 04 01:32:14.745913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.745853 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9"] Nov 04 01:32:14.791234 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:14.791151 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-42958c-67c5787756-sjzj9"] Nov 04 01:32:15.089333 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:15.089255 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:32:16.178572 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.178514 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" containerID="cri-o://50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8" gracePeriod=45 Nov 04 01:32:16.450888 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.450862 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:32:16.507017 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.506989 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf329b96-b5f3-4c89-be83-0e107880e64f" path="/var/lib/kubelet/pods/cf329b96-b5f3-4c89-be83-0e107880e64f/volumes" Nov 04 01:32:16.590756 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.590724 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d846bbb-bee7-4247-ad29-726123c73ed8-kube-api-access-2v7nm\" (UniqueName: \"kubernetes.io/projected/6d846bbb-bee7-4247-ad29-726123c73ed8-kube-api-access-2v7nm\") pod \"6d846bbb-bee7-4247-ad29-726123c73ed8\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " Nov 04 01:32:16.591075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.591049 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-tmp-dir\") pod \"6d846bbb-bee7-4247-ad29-726123c73ed8\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " Nov 04 01:32:16.591307 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.591284 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-dshm\") pod \"6d846bbb-bee7-4247-ad29-726123c73ed8\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " Nov 04 01:32:16.591486 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.591465 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-model-cache\") pod \"6d846bbb-bee7-4247-ad29-726123c73ed8\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " Nov 04 01:32:16.591579 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.591529 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-tmp-dir" pod "6d846bbb-bee7-4247-ad29-726123c73ed8" (UID: "6d846bbb-bee7-4247-ad29-726123c73ed8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:16.591579 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.591571 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-home\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-home\") pod \"6d846bbb-bee7-4247-ad29-726123c73ed8\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " Nov 04 01:32:16.591701 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.591688 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d846bbb-bee7-4247-ad29-726123c73ed8-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d846bbb-bee7-4247-ad29-726123c73ed8-tls-certs\") pod \"6d846bbb-bee7-4247-ad29-726123c73ed8\" (UID: \"6d846bbb-bee7-4247-ad29-726123c73ed8\") " Nov 04 01:32:16.591760 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.591739 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-model-cache" pod "6d846bbb-bee7-4247-ad29-726123c73ed8" (UID: "6d846bbb-bee7-4247-ad29-726123c73ed8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:16.591944 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.591923 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-home" pod "6d846bbb-bee7-4247-ad29-726123c73ed8" (UID: "6d846bbb-bee7-4247-ad29-726123c73ed8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:16.592316 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.592293 5786 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-tmp-dir\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:16.592316 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.592315 5786 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-model-cache\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:16.592480 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.592330 5786 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-home\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:16.596387 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.595169 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d846bbb-bee7-4247-ad29-726123c73ed8-kube-api-access-2v7nm" pod "6d846bbb-bee7-4247-ad29-726123c73ed8" (UID: "6d846bbb-bee7-4247-ad29-726123c73ed8"). InnerVolumeSpecName "kube-api-access-2v7nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:16.596387 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.596172 5786 generic.go:356] "Generic (PLEG): container finished" podID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerID="50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8" exitCode=0 Nov 04 01:32:16.597402 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.596775 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" event={"ID":"6d846bbb-bee7-4247-ad29-726123c73ed8","Type":"ContainerDied","Data":"50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8"} Nov 04 01:32:16.597402 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.596828 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" event={"ID":"6d846bbb-bee7-4247-ad29-726123c73ed8","Type":"ContainerDied","Data":"b596533dfd7d8795a59e444b90988396268c6af228b288f9bc50493bded5740c"} Nov 04 01:32:16.597402 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.596851 5786 scope.go:122] "RemoveContainer" containerID="50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8" Nov 04 01:32:16.597402 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.596884 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt" Nov 04 01:32:16.597402 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.597368 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-dshm" pod "6d846bbb-bee7-4247-ad29-726123c73ed8" (UID: "6d846bbb-bee7-4247-ad29-726123c73ed8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:16.601555 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.601526 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d846bbb-bee7-4247-ad29-726123c73ed8-tls-certs" pod "6d846bbb-bee7-4247-ad29-726123c73ed8" (UID: "6d846bbb-bee7-4247-ad29-726123c73ed8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:16.630129 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.630085 5786 scope.go:122] "RemoveContainer" containerID="50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8" Nov 04 01:32:16.630590 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:16.630563 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8\": container with ID starting with 50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8 not found: ID does not exist" containerID="50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8" Nov 04 01:32:16.630706 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.630604 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8"} err="failed to get container status \"50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8\": rpc error: code = NotFound desc = could not find container \"50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8\": container with ID starting with 50d755cc3961deb187be8a132c7b130818bdd13279934d95f404ff016c64c0f8 not found: ID does not exist" Nov 04 01:32:16.693622 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.693545 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v7nm\" (UniqueName: \"kubernetes.io/projected/6d846bbb-bee7-4247-ad29-726123c73ed8-kube-api-access-2v7nm\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:16.693622 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.693574 5786 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d846bbb-bee7-4247-ad29-726123c73ed8-dshm\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:16.693622 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:16.693583 5786 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d846bbb-bee7-4247-ad29-726123c73ed8-tls-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:17.021746 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:17.021656 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt"] Nov 04 01:32:17.025631 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:17.025604 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-678aaf/distinct-2-w6-678aaf-kserve-599bc6fc5f-tf4jt"] Nov 04 01:32:18.505939 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:18.505905 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" path="/var/lib/kubelet/pods/6d846bbb-bee7-4247-ad29-726123c73ed8/volumes" Nov 04 01:32:28.826629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:28.826593 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq"] Nov 04 01:32:28.949147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:28.949088 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt"] Nov 04 01:32:29.640520 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:29.640489 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4"] Nov 04 01:32:33.827251 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:33.827190 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" podUID="fb97732a-38f0-4cf8-a6eb-44a54a99347b" containerName="payload-processing" containerID="cri-o://36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3" gracePeriod=25 Nov 04 01:32:33.949883 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:33.949848 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" podUID="02e113e2-7693-402f-a137-f37c551d250e" containerName="payload-pre-processing" containerID="cri-o://e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad" gracePeriod=25 Nov 04 01:32:34.093266 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.093199 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" Nov 04 01:32:34.163791 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.163752 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb97732a-38f0-4cf8-a6eb-44a54a99347b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb97732a-38f0-4cf8-a6eb-44a54a99347b-plugins-config-volume\") pod \"fb97732a-38f0-4cf8-a6eb-44a54a99347b\" (UID: \"fb97732a-38f0-4cf8-a6eb-44a54a99347b\") " Nov 04 01:32:34.164040 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.163888 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb97732a-38f0-4cf8-a6eb-44a54a99347b-kube-api-access-n27xd\" (UniqueName: \"kubernetes.io/projected/fb97732a-38f0-4cf8-a6eb-44a54a99347b-kube-api-access-n27xd\") pod \"fb97732a-38f0-4cf8-a6eb-44a54a99347b\" (UID: \"fb97732a-38f0-4cf8-a6eb-44a54a99347b\") " Nov 04 01:32:34.164295 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.164266 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb97732a-38f0-4cf8-a6eb-44a54a99347b-plugins-config-volume" pod "fb97732a-38f0-4cf8-a6eb-44a54a99347b" (UID: "fb97732a-38f0-4cf8-a6eb-44a54a99347b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:34.164586 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.164563 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb97732a-38f0-4cf8-a6eb-44a54a99347b-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:34.166600 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.166575 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb97732a-38f0-4cf8-a6eb-44a54a99347b-kube-api-access-n27xd" pod "fb97732a-38f0-4cf8-a6eb-44a54a99347b" (UID: "fb97732a-38f0-4cf8-a6eb-44a54a99347b"). InnerVolumeSpecName "kube-api-access-n27xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:34.219291 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.219265 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" Nov 04 01:32:34.265468 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.265426 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e113e2-7693-402f-a137-f37c551d250e-kube-api-access-54b2m\" (UniqueName: \"kubernetes.io/projected/02e113e2-7693-402f-a137-f37c551d250e-kube-api-access-54b2m\") pod \"02e113e2-7693-402f-a137-f37c551d250e\" (UID: \"02e113e2-7693-402f-a137-f37c551d250e\") " Nov 04 01:32:34.265687 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.265662 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e113e2-7693-402f-a137-f37c551d250e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e113e2-7693-402f-a137-f37c551d250e-plugins-config-volume\") pod \"02e113e2-7693-402f-a137-f37c551d250e\" (UID: \"02e113e2-7693-402f-a137-f37c551d250e\") " Nov 04 01:32:34.266009 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.265988 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e113e2-7693-402f-a137-f37c551d250e-plugins-config-volume" pod "02e113e2-7693-402f-a137-f37c551d250e" (UID: "02e113e2-7693-402f-a137-f37c551d250e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:34.266199 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.266182 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n27xd\" (UniqueName: \"kubernetes.io/projected/fb97732a-38f0-4cf8-a6eb-44a54a99347b-kube-api-access-n27xd\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:34.266247 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.266203 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e113e2-7693-402f-a137-f37c551d250e-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:34.267667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.267636 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e113e2-7693-402f-a137-f37c551d250e-kube-api-access-54b2m" pod "02e113e2-7693-402f-a137-f37c551d250e" (UID: "02e113e2-7693-402f-a137-f37c551d250e"). InnerVolumeSpecName "kube-api-access-54b2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:34.367890 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.367849 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54b2m\" (UniqueName: \"kubernetes.io/projected/02e113e2-7693-402f-a137-f37c551d250e-kube-api-access-54b2m\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:34.775554 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.775481 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:32:34.830333 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.830289 5786 generic.go:356] "Generic (PLEG): container finished" podID="02e113e2-7693-402f-a137-f37c551d250e" containerID="e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad" exitCode=0 Nov 04 01:32:34.830852 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.830364 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" Nov 04 01:32:34.830852 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.830372 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" event={"ID":"02e113e2-7693-402f-a137-f37c551d250e","Type":"ContainerDied","Data":"e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad"} Nov 04 01:32:34.830852 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.830484 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt" event={"ID":"02e113e2-7693-402f-a137-f37c551d250e","Type":"ContainerDied","Data":"73ea5d8ffea6a29479c7e4d65b70b1a808a8320ae37a5b11467c54d2b7a00378"} Nov 04 01:32:34.830852 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.830518 5786 scope.go:122] "RemoveContainer" containerID="e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad" Nov 04 01:32:34.832044 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.831990 5786 generic.go:356] "Generic (PLEG): container finished" podID="fb97732a-38f0-4cf8-a6eb-44a54a99347b" containerID="36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3" exitCode=0 Nov 04 01:32:34.832135 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.832045 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" Nov 04 01:32:34.832135 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.832077 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" event={"ID":"fb97732a-38f0-4cf8-a6eb-44a54a99347b","Type":"ContainerDied","Data":"36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3"} Nov 04 01:32:34.832249 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.832144 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq" event={"ID":"fb97732a-38f0-4cf8-a6eb-44a54a99347b","Type":"ContainerDied","Data":"06808acd9e7f0b808c72a22519cd630a23d5e44d1bb670f371e317bfe126b307"} Nov 04 01:32:34.853060 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.853037 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:32:34.857541 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.857519 5786 scope.go:122] "RemoveContainer" containerID="e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad" Nov 04 01:32:34.857834 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:34.857807 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad\": container with ID starting with e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad not found: ID does not exist" containerID="e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad" Nov 04 01:32:34.857927 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.857839 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad"} err="failed to get container status \"e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad\": rpc error: code = NotFound desc = could not find container \"e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad\": container with ID starting with e752f7d29f624f0adc92c38e4d726c10b7c4f06100f840fed82b680b5b9e60ad not found: ID does not exist" Nov 04 01:32:34.857927 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.857857 5786 scope.go:122] "RemoveContainer" containerID="36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3" Nov 04 01:32:34.868183 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.868160 5786 scope.go:122] "RemoveContainer" containerID="36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3" Nov 04 01:32:34.868474 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:34.868448 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3\": container with ID starting with 36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3 not found: ID does not exist" containerID="36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3" Nov 04 01:32:34.868534 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.868484 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3"} err="failed to get container status \"36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3\": rpc error: code = NotFound desc = could not find container \"36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3\": container with ID starting with 36ec26b3d6302787dce65eebaab1cffa96fbb1dfdfbcaf9a16ae54d444fd87a3 not found: ID does not exist" Nov 04 01:32:34.996046 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:34.996014 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt"] Nov 04 01:32:35.043662 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:35.043577 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-5778ad-5459f5b799-j69gt"] Nov 04 01:32:35.143399 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:35.143368 5786 scope.go:122] "RemoveContainer" containerID="6328365cb914640034601411e576bf3d86d7fe45f7173dc0788e93f59dff7698" Nov 04 01:32:35.162974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:35.162945 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq"] Nov 04 01:32:35.168851 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:35.168791 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-5778ad-7848d7d5db-dh7rq"] Nov 04 01:32:36.505837 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:36.505805 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e113e2-7693-402f-a137-f37c551d250e" path="/var/lib/kubelet/pods/02e113e2-7693-402f-a137-f37c551d250e/volumes" Nov 04 01:32:36.506279 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:36.506263 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb97732a-38f0-4cf8-a6eb-44a54a99347b" path="/var/lib/kubelet/pods/fb97732a-38f0-4cf8-a6eb-44a54a99347b/volumes" Nov 04 01:32:37.718200 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:37.718166 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk"] Nov 04 01:32:37.718772 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:37.718397 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" podUID="3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8" containerName="maas-api" containerID="cri-o://e50175a88b8985791445f150e32ba2bd63a0af56d877eafe35fb2a92bb5f1f6a" gracePeriod=30 Nov 04 01:32:37.852603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:37.852571 5786 generic.go:356] "Generic (PLEG): container finished" podID="3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8" containerID="e50175a88b8985791445f150e32ba2bd63a0af56d877eafe35fb2a92bb5f1f6a" exitCode=0 Nov 04 01:32:37.852753 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:37.852655 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" event={"ID":"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8","Type":"ContainerDied","Data":"e50175a88b8985791445f150e32ba2bd63a0af56d877eafe35fb2a92bb5f1f6a"} Nov 04 01:32:38.075835 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.075811 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:32:38.154612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.154567 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-openshift-service-ca\") pod \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " Nov 04 01:32:38.155147 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.155094 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-kube-api-access-xgx6m\" (UniqueName: \"kubernetes.io/projected/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-kube-api-access-xgx6m\") pod \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " Nov 04 01:32:38.155620 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.155599 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-metrics-certs\") pod \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " Nov 04 01:32:38.155948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.155927 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-maas-api-tls\") pod \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\" (UID: \"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8\") " Nov 04 01:32:38.172287 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.163870 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-openshift-service-ca" pod "3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8" (UID: "3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:38.172287 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.168522 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-maas-api-tls" pod "3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8" (UID: "3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:38.172536 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.172484 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-kube-api-access-xgx6m" pod "3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8" (UID: "3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8"). InnerVolumeSpecName "kube-api-access-xgx6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:38.172906 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.172871 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-metrics-certs" pod "3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8" (UID: "3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:38.257501 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.257471 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:38.257501 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.257502 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgx6m\" (UniqueName: \"kubernetes.io/projected/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-kube-api-access-xgx6m\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:38.257723 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.257521 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:38.257723 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.257535 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:38.867932 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.867890 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" event={"ID":"3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8","Type":"ContainerDied","Data":"f316c94dc05c69457f8dcba83bb4131b1dcb20140ece693d44d13fa2f744203b"} Nov 04 01:32:38.868490 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.868004 5786 scope.go:122] "RemoveContainer" containerID="e50175a88b8985791445f150e32ba2bd63a0af56d877eafe35fb2a92bb5f1f6a" Nov 04 01:32:38.868490 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:38.868168 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk" Nov 04 01:32:39.027217 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:39.027180 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk"] Nov 04 01:32:39.033497 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:39.033472 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-5778ad-7c4d855775-9f5nk"] Nov 04 01:32:40.504977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:40.504945 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8" path="/var/lib/kubelet/pods/3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8/volumes" Nov 04 01:32:44.662608 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:44.662571 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" containerID="cri-o://8cdf6ea600735d12704bf95b2ea53517941a7f7866b037140f4a659ed1f3128a" gracePeriod=45 Nov 04 01:32:44.690968 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:44.690935 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused" Nov 04 01:32:44.908415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:44.908384 5786 generic.go:356] "Generic (PLEG): container finished" podID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerID="8cdf6ea600735d12704bf95b2ea53517941a7f7866b037140f4a659ed1f3128a" exitCode=0 Nov 04 01:32:44.908615 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:44.908464 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" event={"ID":"cd8c651e-858d-4560-97a5-d3f7f4ce7a21","Type":"ContainerDied","Data":"8cdf6ea600735d12704bf95b2ea53517941a7f7866b037140f4a659ed1f3128a"} Nov 04 01:32:44.951240 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:44.951210 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:32:45.062879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.062847 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-dshm\") pod \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " Nov 04 01:32:45.063026 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.062898 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-model-cache\") pod \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " Nov 04 01:32:45.063026 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.062958 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-home\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-home\") pod \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " Nov 04 01:32:45.063026 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.062981 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tls-certs\") pod \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " Nov 04 01:32:45.063238 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.063034 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tmp-dir\") pod \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " Nov 04 01:32:45.063238 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.063075 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-kube-api-access-94kvx\" (UniqueName: \"kubernetes.io/projected/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-kube-api-access-94kvx\") pod \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\" (UID: \"cd8c651e-858d-4560-97a5-d3f7f4ce7a21\") " Nov 04 01:32:45.063359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.063277 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-home" pod "cd8c651e-858d-4560-97a5-d3f7f4ce7a21" (UID: "cd8c651e-858d-4560-97a5-d3f7f4ce7a21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:45.063359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.063296 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-model-cache" pod "cd8c651e-858d-4560-97a5-d3f7f4ce7a21" (UID: "cd8c651e-858d-4560-97a5-d3f7f4ce7a21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:45.063455 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.063387 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tmp-dir" pod "cd8c651e-858d-4560-97a5-d3f7f4ce7a21" (UID: "cd8c651e-858d-4560-97a5-d3f7f4ce7a21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:45.063807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.063778 5786 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-model-cache\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.063807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.063811 5786 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-home\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.064010 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.063829 5786 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tmp-dir\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.065278 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.065254 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-dshm" pod "cd8c651e-858d-4560-97a5-d3f7f4ce7a21" (UID: "cd8c651e-858d-4560-97a5-d3f7f4ce7a21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:45.065394 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.065350 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tls-certs" pod "cd8c651e-858d-4560-97a5-d3f7f4ce7a21" (UID: "cd8c651e-858d-4560-97a5-d3f7f4ce7a21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:45.065476 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.065454 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-kube-api-access-94kvx" pod "cd8c651e-858d-4560-97a5-d3f7f4ce7a21" (UID: "cd8c651e-858d-4560-97a5-d3f7f4ce7a21"). InnerVolumeSpecName "kube-api-access-94kvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:45.093753 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:45.093703 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:32:45.165546 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.165514 5786 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-tls-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.165546 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.165546 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94kvx\" (UniqueName: \"kubernetes.io/projected/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-kube-api-access-94kvx\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.165748 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.165559 5786 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd8c651e-858d-4560-97a5-d3f7f4ce7a21-dshm\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.916965 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.916934 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" Nov 04 01:32:45.917395 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.916968 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4" event={"ID":"cd8c651e-858d-4560-97a5-d3f7f4ce7a21","Type":"ContainerDied","Data":"35084b0b0ad8c1adaf24da73e095727687c8215d71d0a5cb9d4157135a95b63c"} Nov 04 01:32:45.917395 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:45.917022 5786 scope.go:122] "RemoveContainer" containerID="8cdf6ea600735d12704bf95b2ea53517941a7f7866b037140f4a659ed1f3128a" Nov 04 01:32:46.074680 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:46.074645 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4"] Nov 04 01:32:46.122488 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:46.122454 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/distinct-2-w0-e993d7-kserve-85576d645c-p2xd4"] Nov 04 01:32:46.506114 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:46.506084 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" path="/var/lib/kubelet/pods/cd8c651e-858d-4560-97a5-d3f7f4ce7a21/volumes" Nov 04 01:32:47.522324 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:47.522292 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj"] Nov 04 01:32:47.522740 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:47.522495 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" podUID="3437edcf-befd-4454-b1db-f3ec59979527" containerName="istio-proxy" containerID="cri-o://86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445" gracePeriod=30 Nov 04 01:32:48.941635 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.941605 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-bc2c93"] Nov 04 01:32:48.942228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942153 5786 cpu_manager.go:414] "containerMap: removing container" podUID="f5182da9-f486-453f-80b3-2946aae607fa" containerName="maas-api" Nov 04 01:32:48.942228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942167 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5182da9-f486-453f-80b3-2946aae607fa" containerName="maas-api" Nov 04 01:32:48.942228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942182 5786 cpu_manager.go:414] "containerMap: removing container" podUID="fb97732a-38f0-4cf8-a6eb-44a54a99347b" containerName="payload-processing" Nov 04 01:32:48.942228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942188 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb97732a-38f0-4cf8-a6eb-44a54a99347b" containerName="payload-processing" Nov 04 01:32:48.942228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942195 5786 cpu_manager.go:414] "containerMap: removing container" podUID="02e113e2-7693-402f-a137-f37c551d250e" containerName="payload-pre-processing" Nov 04 01:32:48.942228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942200 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e113e2-7693-402f-a137-f37c551d250e" containerName="payload-pre-processing" Nov 04 01:32:48.942228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942213 5786 cpu_manager.go:414] "containerMap: removing container" podUID="d198877f-a0af-4c3a-8902-160811220bd8" containerName="payload-pre-processing" Nov 04 01:32:48.942228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942218 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d198877f-a0af-4c3a-8902-160811220bd8" containerName="payload-pre-processing" Nov 04 01:32:48.942228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942231 5786 cpu_manager.go:414] "containerMap: removing container" podUID="97d6013a-d49b-4639-88dd-f5e458f50d28" containerName="payload-processing" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942239 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d6013a-d49b-4639-88dd-f5e458f50d28" containerName="payload-processing" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942249 5786 cpu_manager.go:414] "containerMap: removing container" podUID="cf329b96-b5f3-4c89-be83-0e107880e64f" containerName="maas-api" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942256 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf329b96-b5f3-4c89-be83-0e107880e64f" containerName="maas-api" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942267 5786 cpu_manager.go:414] "containerMap: removing container" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942273 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942290 5786 cpu_manager.go:414] "containerMap: removing container" podUID="3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8" containerName="maas-api" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942294 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8" containerName="maas-api" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942311 5786 cpu_manager.go:414] "containerMap: removing container" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942315 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942429 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d846bbb-bee7-4247-ad29-726123c73ed8" containerName="main" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942441 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5182da9-f486-453f-80b3-2946aae607fa" containerName="maas-api" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942446 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb97732a-38f0-4cf8-a6eb-44a54a99347b" containerName="payload-processing" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942453 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e113e2-7693-402f-a137-f37c551d250e" containerName="payload-pre-processing" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942459 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="d198877f-a0af-4c3a-8902-160811220bd8" containerName="payload-pre-processing" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942466 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d6013a-d49b-4639-88dd-f5e458f50d28" containerName="payload-processing" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942472 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf329b96-b5f3-4c89-be83-0e107880e64f" containerName="maas-api" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942478 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8c651e-858d-4560-97a5-d3f7f4ce7a21" containerName="main" Nov 04 01:32:48.942629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.942485 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb0ebe5-687e-4d22-a7c3-5ea8171acdc8" containerName="maas-api" Nov 04 01:32:48.947322 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.947303 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-bc2c93" Nov 04 01:32:48.999154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:48.999105 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-bc2c93"] Nov 04 01:32:49.032129 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:49.032093 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbqr\" (UniqueName: \"kubernetes.io/projected/0a671648-a142-4e86-bad3-9f6071b32286-kube-api-access-kbbqr\") pod \"test-curl-7240-bc2c93\" (UID: \"0a671648-a142-4e86-bad3-9f6071b32286\") " pod="opendatahub/test-curl-7240-bc2c93" Nov 04 01:32:49.043886 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:49.043861 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bqc6h\"" Nov 04 01:32:49.133568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:49.133533 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbqr\" (UniqueName: \"kubernetes.io/projected/0a671648-a142-4e86-bad3-9f6071b32286-kube-api-access-kbbqr\") pod \"test-curl-7240-bc2c93\" (UID: \"0a671648-a142-4e86-bad3-9f6071b32286\") " pod="opendatahub/test-curl-7240-bc2c93" Nov 04 01:32:49.190340 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:49.190307 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbqr\" (UniqueName: \"kubernetes.io/projected/0a671648-a142-4e86-bad3-9f6071b32286-kube-api-access-kbbqr\") pod \"test-curl-7240-bc2c93\" (UID: \"0a671648-a142-4e86-bad3-9f6071b32286\") " pod="opendatahub/test-curl-7240-bc2c93" Nov 04 01:32:49.259662 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:49.259591 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-bc2c93" Nov 04 01:32:49.644761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:49.644734 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-bc2c93"] Nov 04 01:32:49.648079 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:32:49.648048 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a671648_a142_4e86_bad3_9f6071b32286.slice/crio-9b27d414fd5cd5c2b6e03e76fa443669040a869754338a15b9831a55ef450e6a WatchSource:0}: Error finding container 9b27d414fd5cd5c2b6e03e76fa443669040a869754338a15b9831a55ef450e6a: Status 404 returned error can't find the container with id 9b27d414fd5cd5c2b6e03e76fa443669040a869754338a15b9831a55ef450e6a Nov 04 01:32:49.941765 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:49.941676 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-bc2c93" event={"ID":"0a671648-a142-4e86-bad3-9f6071b32286","Type":"ContainerStarted","Data":"1499e4ecad878ebfe61378d86dd244b3a19f9f52694422e4cfa2db7793bbfa94"} Nov 04 01:32:49.941765 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:49.941721 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-bc2c93" event={"ID":"0a671648-a142-4e86-bad3-9f6071b32286","Type":"ContainerStarted","Data":"9b27d414fd5cd5c2b6e03e76fa443669040a869754338a15b9831a55ef450e6a"} Nov 04 01:32:50.050977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:50.050923 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-bc2c93" podStartSLOduration=2.05090892 podStartE2EDuration="2.05090892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:50.050188656 +0000 UTC m=+2115.666534416" watchObservedRunningTime="2026-11-04 01:32:50.05090892 +0000 UTC m=+2115.667254684" Nov 04 01:32:51.566140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:51.565317 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-bc2c93"] Nov 04 01:32:51.570019 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:51.569980 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-bc2c93"] Nov 04 01:32:51.717606 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:51.717571 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-4df8db"] Nov 04 01:32:51.718206 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:51.718188 5786 cpu_manager.go:414] "containerMap: removing container" podUID="0a671648-a142-4e86-bad3-9f6071b32286" containerName="test-curl-7240-bc2c93" Nov 04 01:32:51.718279 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:51.718206 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a671648-a142-4e86-bad3-9f6071b32286" containerName="test-curl-7240-bc2c93" Nov 04 01:32:51.718355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:51.718319 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a671648-a142-4e86-bad3-9f6071b32286" containerName="test-curl-7240-bc2c93" Nov 04 01:32:51.722286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:51.722266 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-4df8db" Nov 04 01:32:51.773529 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:51.773483 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvj2r\" (UniqueName: \"kubernetes.io/projected/a0dc06ca-7249-49bd-9946-a5ccd3f55793-kube-api-access-bvj2r\") pod \"test-curl-7240-4df8db\" (UID: \"a0dc06ca-7249-49bd-9946-a5ccd3f55793\") " pod="opendatahub/test-curl-7240-4df8db" Nov 04 01:32:51.777340 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:51.777299 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-4df8db"] Nov 04 01:32:51.875395 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:51.875363 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvj2r\" (UniqueName: \"kubernetes.io/projected/a0dc06ca-7249-49bd-9946-a5ccd3f55793-kube-api-access-bvj2r\") pod \"test-curl-7240-4df8db\" (UID: \"a0dc06ca-7249-49bd-9946-a5ccd3f55793\") " pod="opendatahub/test-curl-7240-4df8db" Nov 04 01:32:51.931495 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:51.931462 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvj2r\" (UniqueName: \"kubernetes.io/projected/a0dc06ca-7249-49bd-9946-a5ccd3f55793-kube-api-access-bvj2r\") pod \"test-curl-7240-4df8db\" (UID: \"a0dc06ca-7249-49bd-9946-a5ccd3f55793\") " pod="opendatahub/test-curl-7240-4df8db" Nov 04 01:32:51.960786 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:51.960728 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-bc2c93" podUID="0a671648-a142-4e86-bad3-9f6071b32286" containerName="test-curl-7240-bc2c93" containerID="cri-o://1499e4ecad878ebfe61378d86dd244b3a19f9f52694422e4cfa2db7793bbfa94" gracePeriod=2 Nov 04 01:32:52.011802 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:52.011760 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-bc2c93\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="0a671648-a142-4e86-bad3-9f6071b32286" pod="opendatahub/test-curl-7240-bc2c93" Nov 04 01:32:52.036664 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.036626 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-4df8db" Nov 04 01:32:52.410265 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.410227 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-4df8db"] Nov 04 01:32:52.416143 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:32:52.416078 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0dc06ca_7249_49bd_9946_a5ccd3f55793.slice/crio-bdeaea9af579755802ff24e0bda6ca3949b9746d0c5747370c7f01defac83f7a WatchSource:0}: Error finding container bdeaea9af579755802ff24e0bda6ca3949b9746d0c5747370c7f01defac83f7a: Status 404 returned error can't find the container with id bdeaea9af579755802ff24e0bda6ca3949b9746d0c5747370c7f01defac83f7a Nov 04 01:32:52.783547 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.783522 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:32:52.895947 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.895914 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-data\") pod \"3437edcf-befd-4454-b1db-f3ec59979527\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " Nov 04 01:32:52.895947 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.895954 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-envoy\") pod \"3437edcf-befd-4454-b1db-f3ec59979527\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " Nov 04 01:32:52.896189 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.895993 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-certs\") pod \"3437edcf-befd-4454-b1db-f3ec59979527\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " Nov 04 01:32:52.896189 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896064 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-kube-api-access-pcn6d\" (UniqueName: \"kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-kube-api-access-pcn6d\") pod \"3437edcf-befd-4454-b1db-f3ec59979527\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " Nov 04 01:32:52.896189 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896110 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-istio-token\" (UniqueName: \"kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-istio-token\") pod \"3437edcf-befd-4454-b1db-f3ec59979527\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " Nov 04 01:32:52.896364 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896200 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-socket\") pod \"3437edcf-befd-4454-b1db-f3ec59979527\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " Nov 04 01:32:52.896364 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896212 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-data" pod "3437edcf-befd-4454-b1db-f3ec59979527" (UID: "3437edcf-befd-4454-b1db-f3ec59979527"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:52.896364 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896234 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3437edcf-befd-4454-b1db-f3ec59979527-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3437edcf-befd-4454-b1db-f3ec59979527-istiod-ca-cert\") pod \"3437edcf-befd-4454-b1db-f3ec59979527\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " Nov 04 01:32:52.896364 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896254 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3437edcf-befd-4454-b1db-f3ec59979527-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3437edcf-befd-4454-b1db-f3ec59979527-istio-podinfo\") pod \"3437edcf-befd-4454-b1db-f3ec59979527\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " Nov 04 01:32:52.896364 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896325 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-credential-socket\") pod \"3437edcf-befd-4454-b1db-f3ec59979527\" (UID: \"3437edcf-befd-4454-b1db-f3ec59979527\") " Nov 04 01:32:52.896616 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896495 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-socket" pod "3437edcf-befd-4454-b1db-f3ec59979527" (UID: "3437edcf-befd-4454-b1db-f3ec59979527"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:52.896866 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896671 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3437edcf-befd-4454-b1db-f3ec59979527-istiod-ca-cert" pod "3437edcf-befd-4454-b1db-f3ec59979527" (UID: "3437edcf-befd-4454-b1db-f3ec59979527"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:52.896866 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896764 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-credential-socket" pod "3437edcf-befd-4454-b1db-f3ec59979527" (UID: "3437edcf-befd-4454-b1db-f3ec59979527"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:52.896866 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896803 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.896866 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896826 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.896866 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.896861 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-certs" pod "3437edcf-befd-4454-b1db-f3ec59979527" (UID: "3437edcf-befd-4454-b1db-f3ec59979527"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:52.898462 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.898438 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3437edcf-befd-4454-b1db-f3ec59979527-istio-podinfo" pod "3437edcf-befd-4454-b1db-f3ec59979527" (UID: "3437edcf-befd-4454-b1db-f3ec59979527"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:52.898558 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.898466 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-istio-token" pod "3437edcf-befd-4454-b1db-f3ec59979527" (UID: "3437edcf-befd-4454-b1db-f3ec59979527"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:52.898632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.898618 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-envoy" pod "3437edcf-befd-4454-b1db-f3ec59979527" (UID: "3437edcf-befd-4454-b1db-f3ec59979527"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:52.898725 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.898710 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-kube-api-access-pcn6d" pod "3437edcf-befd-4454-b1db-f3ec59979527" (UID: "3437edcf-befd-4454-b1db-f3ec59979527"). InnerVolumeSpecName "kube-api-access-pcn6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:52.969330 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.969233 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-4df8db" event={"ID":"a0dc06ca-7249-49bd-9946-a5ccd3f55793","Type":"ContainerStarted","Data":"587aaf639af2a8e038525515d7f58ffc71dad621e96bc2ecf6fe200bffd770c0"} Nov 04 01:32:52.969330 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.969286 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-4df8db" event={"ID":"a0dc06ca-7249-49bd-9946-a5ccd3f55793","Type":"ContainerStarted","Data":"bdeaea9af579755802ff24e0bda6ca3949b9746d0c5747370c7f01defac83f7a"} Nov 04 01:32:52.970960 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.970932 5786 generic.go:356] "Generic (PLEG): container finished" podID="3437edcf-befd-4454-b1db-f3ec59979527" containerID="86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445" exitCode=0 Nov 04 01:32:52.971069 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.970976 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" Nov 04 01:32:52.971069 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.971007 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" event={"ID":"3437edcf-befd-4454-b1db-f3ec59979527","Type":"ContainerDied","Data":"86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445"} Nov 04 01:32:52.971069 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.971056 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj" event={"ID":"3437edcf-befd-4454-b1db-f3ec59979527","Type":"ContainerDied","Data":"a4318f55ba5aa1abee8c22875132c6ef8542f19f54abd92322cbd9f2bf3abd3c"} Nov 04 01:32:52.971216 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.971078 5786 scope.go:122] "RemoveContainer" containerID="86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445" Nov 04 01:32:52.981676 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.981657 5786 scope.go:122] "RemoveContainer" containerID="0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8" Nov 04 01:32:52.990415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.990390 5786 scope.go:122] "RemoveContainer" containerID="86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445" Nov 04 01:32:52.990696 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:52.990675 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445\": container with ID starting with 86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445 not found: ID does not exist" containerID="86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445" Nov 04 01:32:52.990740 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.990706 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445"} err="failed to get container status \"86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445\": rpc error: code = NotFound desc = could not find container \"86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445\": container with ID starting with 86399c3f77687bf71924b2a1d62e19fc86a976228fde9a1c7f650597225ce445 not found: ID does not exist" Nov 04 01:32:52.990740 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.990723 5786 scope.go:122] "RemoveContainer" containerID="0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8" Nov 04 01:32:52.990994 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:52.990976 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8\": container with ID starting with 0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8 not found: ID does not exist" containerID="0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8" Nov 04 01:32:52.991031 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.991002 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8"} err="failed to get container status \"0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8\": rpc error: code = NotFound desc = could not find container \"0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8\": container with ID starting with 0f3d0497f97c77f73f9133960563337c14a219bc3810637533d76fe0eb3602a8 not found: ID does not exist" Nov 04 01:32:52.998417 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.998388 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcn6d\" (UniqueName: \"kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-kube-api-access-pcn6d\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.998503 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.998424 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3437edcf-befd-4454-b1db-f3ec59979527-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.998503 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.998435 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3437edcf-befd-4454-b1db-f3ec59979527-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.998503 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.998444 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3437edcf-befd-4454-b1db-f3ec59979527-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.998503 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.998453 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.998503 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.998462 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.998503 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:52.998470 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3437edcf-befd-4454-b1db-f3ec59979527-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.124146 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:53.124085 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj"] Nov 04 01:32:53.140676 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:53.140643 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-5778ad-openshift-default-6f7c564c4f-7wxtj"] Nov 04 01:32:53.257383 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:53.257264 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-4df8db" podStartSLOduration=2.257244651 podStartE2EDuration="2.257244651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:53.255749734 +0000 UTC m=+2118.872095497" watchObservedRunningTime="2026-11-04 01:32:53.257244651 +0000 UTC m=+2118.873590418" Nov 04 01:32:54.203497 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.203474 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-bc2c93" Nov 04 01:32:54.314044 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.313963 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a671648-a142-4e86-bad3-9f6071b32286-kube-api-access-kbbqr\" (UniqueName: \"kubernetes.io/projected/0a671648-a142-4e86-bad3-9f6071b32286-kube-api-access-kbbqr\") pod \"0a671648-a142-4e86-bad3-9f6071b32286\" (UID: \"0a671648-a142-4e86-bad3-9f6071b32286\") " Nov 04 01:32:54.315996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.315972 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a671648-a142-4e86-bad3-9f6071b32286-kube-api-access-kbbqr" pod "0a671648-a142-4e86-bad3-9f6071b32286" (UID: "0a671648-a142-4e86-bad3-9f6071b32286"). InnerVolumeSpecName "kube-api-access-kbbqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:54.415284 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.415251 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbbqr\" (UniqueName: \"kubernetes.io/projected/0a671648-a142-4e86-bad3-9f6071b32286-kube-api-access-kbbqr\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:54.505575 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.505540 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a671648-a142-4e86-bad3-9f6071b32286" path="/var/lib/kubelet/pods/0a671648-a142-4e86-bad3-9f6071b32286/volumes" Nov 04 01:32:54.506009 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.505987 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3437edcf-befd-4454-b1db-f3ec59979527" path="/var/lib/kubelet/pods/3437edcf-befd-4454-b1db-f3ec59979527/volumes" Nov 04 01:32:54.786002 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.785967 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-4df8db"] Nov 04 01:32:54.791703 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.791679 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-4df8db"] Nov 04 01:32:54.945712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.945681 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-ae65af"] Nov 04 01:32:54.946246 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.946232 5786 cpu_manager.go:414] "containerMap: removing container" podUID="3437edcf-befd-4454-b1db-f3ec59979527" containerName="istio-proxy" Nov 04 01:32:54.946286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.946247 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3437edcf-befd-4454-b1db-f3ec59979527" containerName="istio-proxy" Nov 04 01:32:54.946286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.946256 5786 cpu_manager.go:414] "containerMap: removing container" podUID="3437edcf-befd-4454-b1db-f3ec59979527" containerName="istio-proxy" Nov 04 01:32:54.946286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.946261 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3437edcf-befd-4454-b1db-f3ec59979527" containerName="istio-proxy" Nov 04 01:32:54.946400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.946291 5786 cpu_manager.go:414] "containerMap: removing container" podUID="a0dc06ca-7249-49bd-9946-a5ccd3f55793" containerName="test-curl-7240-4df8db" Nov 04 01:32:54.946400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.946296 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0dc06ca-7249-49bd-9946-a5ccd3f55793" containerName="test-curl-7240-4df8db" Nov 04 01:32:54.946474 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.946425 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="3437edcf-befd-4454-b1db-f3ec59979527" containerName="istio-proxy" Nov 04 01:32:54.946474 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.946437 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="3437edcf-befd-4454-b1db-f3ec59979527" containerName="istio-proxy" Nov 04 01:32:54.946474 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.946457 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0dc06ca-7249-49bd-9946-a5ccd3f55793" containerName="test-curl-7240-4df8db" Nov 04 01:32:54.949909 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:54.949890 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-ae65af" Nov 04 01:32:55.060204 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.048540 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwqh\" (UniqueName: \"kubernetes.io/projected/1b92d1af-03c1-46c3-acf3-078738dbb301-kube-api-access-ptwqh\") pod \"test-curl-7240-ae65af\" (UID: \"1b92d1af-03c1-46c3-acf3-078738dbb301\") " pod="opendatahub/test-curl-7240-ae65af" Nov 04 01:32:55.088169 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.079218 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-ae65af"] Nov 04 01:32:55.099261 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.099198 5786 generic.go:356] "Generic (PLEG): container finished" podID="0a671648-a142-4e86-bad3-9f6071b32286" containerID="1499e4ecad878ebfe61378d86dd244b3a19f9f52694422e4cfa2db7793bbfa94" exitCode=137 Nov 04 01:32:55.099760 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.099599 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-bc2c93" Nov 04 01:32:55.103407 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.099604 5786 scope.go:122] "RemoveContainer" containerID="1499e4ecad878ebfe61378d86dd244b3a19f9f52694422e4cfa2db7793bbfa94" Nov 04 01:32:55.105578 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.105536 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-4df8db" podUID="a0dc06ca-7249-49bd-9946-a5ccd3f55793" containerName="test-curl-7240-4df8db" containerID="cri-o://587aaf639af2a8e038525515d7f58ffc71dad621e96bc2ecf6fe200bffd770c0" gracePeriod=2 Nov 04 01:32:55.119938 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.119897 5786 scope.go:122] "RemoveContainer" containerID="1499e4ecad878ebfe61378d86dd244b3a19f9f52694422e4cfa2db7793bbfa94" Nov 04 01:32:55.120374 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:55.120336 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1499e4ecad878ebfe61378d86dd244b3a19f9f52694422e4cfa2db7793bbfa94\": container with ID starting with 1499e4ecad878ebfe61378d86dd244b3a19f9f52694422e4cfa2db7793bbfa94 not found: ID does not exist" containerID="1499e4ecad878ebfe61378d86dd244b3a19f9f52694422e4cfa2db7793bbfa94" Nov 04 01:32:55.120495 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.120379 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1499e4ecad878ebfe61378d86dd244b3a19f9f52694422e4cfa2db7793bbfa94"} err="failed to get container status \"1499e4ecad878ebfe61378d86dd244b3a19f9f52694422e4cfa2db7793bbfa94\": rpc error: code = NotFound desc = could not find container \"1499e4ecad878ebfe61378d86dd244b3a19f9f52694422e4cfa2db7793bbfa94\": container with ID starting with 1499e4ecad878ebfe61378d86dd244b3a19f9f52694422e4cfa2db7793bbfa94 not found: ID does not exist" Nov 04 01:32:55.149751 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.149707 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwqh\" (UniqueName: \"kubernetes.io/projected/1b92d1af-03c1-46c3-acf3-078738dbb301-kube-api-access-ptwqh\") pod \"test-curl-7240-ae65af\" (UID: \"1b92d1af-03c1-46c3-acf3-078738dbb301\") " pod="opendatahub/test-curl-7240-ae65af" Nov 04 01:32:55.155337 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:55.155304 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-4df8db\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="a0dc06ca-7249-49bd-9946-a5ccd3f55793" pod="opendatahub/test-curl-7240-4df8db" Nov 04 01:32:55.204031 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:55.203995 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-4df8db\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="a0dc06ca-7249-49bd-9946-a5ccd3f55793" pod="opendatahub/test-curl-7240-4df8db" Nov 04 01:32:55.205044 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.205020 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwqh\" (UniqueName: \"kubernetes.io/projected/1b92d1af-03c1-46c3-acf3-078738dbb301-kube-api-access-ptwqh\") pod \"test-curl-7240-ae65af\" (UID: \"1b92d1af-03c1-46c3-acf3-078738dbb301\") " pod="opendatahub/test-curl-7240-ae65af" Nov 04 01:32:55.216830 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.216794 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:32:55.290986 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.290958 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-ae65af" Nov 04 01:32:55.683191 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:55.682235 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-ae65af"] Nov 04 01:32:56.113513 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:56.113483 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-ae65af" event={"ID":"1b92d1af-03c1-46c3-acf3-078738dbb301","Type":"ContainerStarted","Data":"fb3e2c58316fcfb16fd0e9a42f4b720e127509a39d4feb682a860a32d27f384b"} Nov 04 01:32:56.113702 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:56.113522 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-ae65af" event={"ID":"1b92d1af-03c1-46c3-acf3-078738dbb301","Type":"ContainerStarted","Data":"a074aedf6081f14337a4cb164c5842a94035d43889a5d50c7eabb9fa9239e6f3"} Nov 04 01:32:56.162609 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:56.162567 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-4df8db\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="a0dc06ca-7249-49bd-9946-a5ccd3f55793" pod="opendatahub/test-curl-7240-4df8db" Nov 04 01:32:56.272771 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:56.272716 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-ae65af" podStartSLOduration=2.272702729 podStartE2EDuration="2.272702729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:56.270504055 +0000 UTC m=+2121.886849819" watchObservedRunningTime="2026-11-04 01:32:56.272702729 +0000 UTC m=+2121.889048491" Nov 04 01:32:57.316782 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.316755 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-4df8db" Nov 04 01:32:57.401400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.401319 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dc06ca-7249-49bd-9946-a5ccd3f55793-kube-api-access-bvj2r\" (UniqueName: \"kubernetes.io/projected/a0dc06ca-7249-49bd-9946-a5ccd3f55793-kube-api-access-bvj2r\") pod \"a0dc06ca-7249-49bd-9946-a5ccd3f55793\" (UID: \"a0dc06ca-7249-49bd-9946-a5ccd3f55793\") " Nov 04 01:32:57.403463 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.403436 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dc06ca-7249-49bd-9946-a5ccd3f55793-kube-api-access-bvj2r" pod "a0dc06ca-7249-49bd-9946-a5ccd3f55793" (UID: "a0dc06ca-7249-49bd-9946-a5ccd3f55793"). InnerVolumeSpecName "kube-api-access-bvj2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:57.502915 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.502882 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvj2r\" (UniqueName: \"kubernetes.io/projected/a0dc06ca-7249-49bd-9946-a5ccd3f55793-kube-api-access-bvj2r\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:57.675827 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.675206 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-ae65af"] Nov 04 01:32:57.679346 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.679314 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-ae65af"] Nov 04 01:32:57.818748 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.818719 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-7b26ca"] Nov 04 01:32:57.819340 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.819306 5786 cpu_manager.go:414] "containerMap: removing container" podUID="1b92d1af-03c1-46c3-acf3-078738dbb301" containerName="test-curl-7240-ae65af" Nov 04 01:32:57.819340 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.819323 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b92d1af-03c1-46c3-acf3-078738dbb301" containerName="test-curl-7240-ae65af" Nov 04 01:32:57.819471 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.819409 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b92d1af-03c1-46c3-acf3-078738dbb301" containerName="test-curl-7240-ae65af" Nov 04 01:32:57.822764 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.822746 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-7b26ca" Nov 04 01:32:57.873571 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.873546 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-7b26ca"] Nov 04 01:32:57.907594 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:57.907553 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czq62\" (UniqueName: \"kubernetes.io/projected/8ece2c47-d09b-4355-8b9d-7df3a80de475-kube-api-access-czq62\") pod \"test-curl-7240-7b26ca\" (UID: \"8ece2c47-d09b-4355-8b9d-7df3a80de475\") " pod="opendatahub/test-curl-7240-7b26ca" Nov 04 01:32:58.008579 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:58.008494 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czq62\" (UniqueName: \"kubernetes.io/projected/8ece2c47-d09b-4355-8b9d-7df3a80de475-kube-api-access-czq62\") pod \"test-curl-7240-7b26ca\" (UID: \"8ece2c47-d09b-4355-8b9d-7df3a80de475\") " pod="opendatahub/test-curl-7240-7b26ca" Nov 04 01:32:58.065184 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:58.065152 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czq62\" (UniqueName: \"kubernetes.io/projected/8ece2c47-d09b-4355-8b9d-7df3a80de475-kube-api-access-czq62\") pod \"test-curl-7240-7b26ca\" (UID: \"8ece2c47-d09b-4355-8b9d-7df3a80de475\") " pod="opendatahub/test-curl-7240-7b26ca" Nov 04 01:32:58.127587 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:58.127553 5786 generic.go:356] "Generic (PLEG): container finished" podID="a0dc06ca-7249-49bd-9946-a5ccd3f55793" containerID="587aaf639af2a8e038525515d7f58ffc71dad621e96bc2ecf6fe200bffd770c0" exitCode=137 Nov 04 01:32:58.127765 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:58.127601 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-4df8db" Nov 04 01:32:58.127765 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:58.127649 5786 scope.go:122] "RemoveContainer" containerID="587aaf639af2a8e038525515d7f58ffc71dad621e96bc2ecf6fe200bffd770c0" Nov 04 01:32:58.127869 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:58.127813 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-ae65af" podUID="1b92d1af-03c1-46c3-acf3-078738dbb301" containerName="test-curl-7240-ae65af" containerID="cri-o://fb3e2c58316fcfb16fd0e9a42f4b720e127509a39d4feb682a860a32d27f384b" gracePeriod=2 Nov 04 01:32:58.135237 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:58.135217 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-7b26ca" Nov 04 01:32:58.137055 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:58.137038 5786 scope.go:122] "RemoveContainer" containerID="587aaf639af2a8e038525515d7f58ffc71dad621e96bc2ecf6fe200bffd770c0" Nov 04 01:32:58.137342 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:58.137323 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587aaf639af2a8e038525515d7f58ffc71dad621e96bc2ecf6fe200bffd770c0\": container with ID starting with 587aaf639af2a8e038525515d7f58ffc71dad621e96bc2ecf6fe200bffd770c0 not found: ID does not exist" containerID="587aaf639af2a8e038525515d7f58ffc71dad621e96bc2ecf6fe200bffd770c0" Nov 04 01:32:58.137402 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:58.137352 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587aaf639af2a8e038525515d7f58ffc71dad621e96bc2ecf6fe200bffd770c0"} err="failed to get container status \"587aaf639af2a8e038525515d7f58ffc71dad621e96bc2ecf6fe200bffd770c0\": rpc error: code = NotFound desc = could not find container \"587aaf639af2a8e038525515d7f58ffc71dad621e96bc2ecf6fe200bffd770c0\": container with ID starting with 587aaf639af2a8e038525515d7f58ffc71dad621e96bc2ecf6fe200bffd770c0 not found: ID does not exist" Nov 04 01:32:58.177001 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:58.176965 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-ae65af\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="1b92d1af-03c1-46c3-acf3-078738dbb301" pod="opendatahub/test-curl-7240-ae65af" Nov 04 01:32:58.226089 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:58.226055 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-4df8db\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="a0dc06ca-7249-49bd-9946-a5ccd3f55793" pod="opendatahub/test-curl-7240-4df8db" Nov 04 01:32:58.274841 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:32:58.274762 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-ae65af\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="1b92d1af-03c1-46c3-acf3-078738dbb301" pod="opendatahub/test-curl-7240-ae65af" Nov 04 01:32:58.506397 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:58.506367 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0dc06ca-7249-49bd-9946-a5ccd3f55793" path="/var/lib/kubelet/pods/a0dc06ca-7249-49bd-9946-a5ccd3f55793/volumes" Nov 04 01:32:58.507840 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:58.507816 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-7b26ca"] Nov 04 01:32:58.512539 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:32:58.512516 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ece2c47_d09b_4355_8b9d_7df3a80de475.slice/crio-e23272a374926fbf224b304ea085943e683747cae748917f86729b41d9beff78 WatchSource:0}: Error finding container e23272a374926fbf224b304ea085943e683747cae748917f86729b41d9beff78: Status 404 returned error can't find the container with id e23272a374926fbf224b304ea085943e683747cae748917f86729b41d9beff78 Nov 04 01:32:59.134897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:59.134863 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-7b26ca" event={"ID":"8ece2c47-d09b-4355-8b9d-7df3a80de475","Type":"ContainerStarted","Data":"f731eeaca63acdac2187872aafc9e6b8e9b54c9f6baf8bc6ec024692299196bf"} Nov 04 01:32:59.134897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:59.134908 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-7b26ca" event={"ID":"8ece2c47-d09b-4355-8b9d-7df3a80de475","Type":"ContainerStarted","Data":"e23272a374926fbf224b304ea085943e683747cae748917f86729b41d9beff78"} Nov 04 01:32:59.243492 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:59.243434 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-7b26ca" podStartSLOduration=2.243422474 podStartE2EDuration="2.243422474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:59.242909528 +0000 UTC m=+2124.859255291" watchObservedRunningTime="2026-11-04 01:32:59.243422474 +0000 UTC m=+2124.859768237" Nov 04 01:32:59.688228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:59.688193 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz"] Nov 04 01:32:59.688716 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:59.688457 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" containerID="cri-o://c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098" gracePeriod=30 Nov 04 01:32:59.690323 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:59.690301 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2"] Nov 04 01:32:59.690516 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:59.690481 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" containerID="cri-o://ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832" gracePeriod=30 Nov 04 01:32:59.730974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:59.730937 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr"] Nov 04 01:32:59.731220 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:32:59.731193 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" containerID="cri-o://4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0" gracePeriod=30 Nov 04 01:33:00.074857 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:00.074777 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv"] Nov 04 01:33:00.115365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:00.115329 5786 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:00.115505 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:00.115385 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:00.196070 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:00.196037 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d"] Nov 04 01:33:00.342872 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:00.342811 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-ae65af" Nov 04 01:33:00.460044 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:00.460012 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b92d1af-03c1-46c3-acf3-078738dbb301-kube-api-access-ptwqh\" (UniqueName: \"kubernetes.io/projected/1b92d1af-03c1-46c3-acf3-078738dbb301-kube-api-access-ptwqh\") pod \"1b92d1af-03c1-46c3-acf3-078738dbb301\" (UID: \"1b92d1af-03c1-46c3-acf3-078738dbb301\") " Nov 04 01:33:00.462154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:00.462099 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b92d1af-03c1-46c3-acf3-078738dbb301-kube-api-access-ptwqh" pod "1b92d1af-03c1-46c3-acf3-078738dbb301" (UID: "1b92d1af-03c1-46c3-acf3-078738dbb301"). InnerVolumeSpecName "kube-api-access-ptwqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:00.558044 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:00.554645 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b92d1af-03c1-46c3-acf3-078738dbb301" path="/var/lib/kubelet/pods/1b92d1af-03c1-46c3-acf3-078738dbb301/volumes" Nov 04 01:33:00.565383 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:00.565354 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptwqh\" (UniqueName: \"kubernetes.io/projected/1b92d1af-03c1-46c3-acf3-078738dbb301-kube-api-access-ptwqh\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:00.613344 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:00.613315 5786 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b92d1af_03c1_46c3_acf3_078738dbb301.slice/crio-a074aedf6081f14337a4cb164c5842a94035d43889a5d50c7eabb9fa9239e6f3\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b92d1af_03c1_46c3_acf3_078738dbb301.slice\": RecentStats: unable to find data in memory cache]" Nov 04 01:33:00.877293 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:00.877200 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-7b26ca"] Nov 04 01:33:00.891027 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:00.890985 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-7b26ca"] Nov 04 01:33:01.151510 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:01.151425 5786 generic.go:356] "Generic (PLEG): container finished" podID="1b92d1af-03c1-46c3-acf3-078738dbb301" containerID="fb3e2c58316fcfb16fd0e9a42f4b720e127509a39d4feb682a860a32d27f384b" exitCode=137 Nov 04 01:33:01.151510 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:01.151469 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-ae65af" Nov 04 01:33:01.151737 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:01.151519 5786 scope.go:122] "RemoveContainer" containerID="fb3e2c58316fcfb16fd0e9a42f4b720e127509a39d4feb682a860a32d27f384b" Nov 04 01:33:01.151794 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:01.151766 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-7b26ca" podUID="8ece2c47-d09b-4355-8b9d-7df3a80de475" containerName="test-curl-7240-7b26ca" containerID="cri-o://f731eeaca63acdac2187872aafc9e6b8e9b54c9f6baf8bc6ec024692299196bf" gracePeriod=2 Nov 04 01:33:01.160276 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:01.160260 5786 scope.go:122] "RemoveContainer" containerID="fb3e2c58316fcfb16fd0e9a42f4b720e127509a39d4feb682a860a32d27f384b" Nov 04 01:33:01.160554 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:01.160536 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3e2c58316fcfb16fd0e9a42f4b720e127509a39d4feb682a860a32d27f384b\": container with ID starting with fb3e2c58316fcfb16fd0e9a42f4b720e127509a39d4feb682a860a32d27f384b not found: ID does not exist" containerID="fb3e2c58316fcfb16fd0e9a42f4b720e127509a39d4feb682a860a32d27f384b" Nov 04 01:33:01.160632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:01.160562 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3e2c58316fcfb16fd0e9a42f4b720e127509a39d4feb682a860a32d27f384b"} err="failed to get container status \"fb3e2c58316fcfb16fd0e9a42f4b720e127509a39d4feb682a860a32d27f384b\": rpc error: code = NotFound desc = could not find container \"fb3e2c58316fcfb16fd0e9a42f4b720e127509a39d4feb682a860a32d27f384b\": container with ID starting with fb3e2c58316fcfb16fd0e9a42f4b720e127509a39d4feb682a860a32d27f384b not found: ID does not exist" Nov 04 01:33:01.200538 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:01.200505 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-7b26ca\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="8ece2c47-d09b-4355-8b9d-7df3a80de475" pod="opendatahub/test-curl-7240-7b26ca" Nov 04 01:33:01.249529 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:01.249497 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-7b26ca\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="8ece2c47-d09b-4355-8b9d-7df3a80de475" pod="opendatahub/test-curl-7240-7b26ca" Nov 04 01:33:03.415762 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:03.415694 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-7b26ca" Nov 04 01:33:03.519605 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:03.519573 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ece2c47-d09b-4355-8b9d-7df3a80de475-kube-api-access-czq62\" (UniqueName: \"kubernetes.io/projected/8ece2c47-d09b-4355-8b9d-7df3a80de475-kube-api-access-czq62\") pod \"8ece2c47-d09b-4355-8b9d-7df3a80de475\" (UID: \"8ece2c47-d09b-4355-8b9d-7df3a80de475\") " Nov 04 01:33:03.521679 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:03.521645 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ece2c47-d09b-4355-8b9d-7df3a80de475-kube-api-access-czq62" pod "8ece2c47-d09b-4355-8b9d-7df3a80de475" (UID: "8ece2c47-d09b-4355-8b9d-7df3a80de475"). InnerVolumeSpecName "kube-api-access-czq62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:03.620835 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:03.620796 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czq62\" (UniqueName: \"kubernetes.io/projected/8ece2c47-d09b-4355-8b9d-7df3a80de475-kube-api-access-czq62\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:04.172200 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:04.172164 5786 generic.go:356] "Generic (PLEG): container finished" podID="8ece2c47-d09b-4355-8b9d-7df3a80de475" containerID="f731eeaca63acdac2187872aafc9e6b8e9b54c9f6baf8bc6ec024692299196bf" exitCode=137 Nov 04 01:33:04.172375 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:04.172211 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-7b26ca" Nov 04 01:33:04.172375 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:04.172266 5786 scope.go:122] "RemoveContainer" containerID="f731eeaca63acdac2187872aafc9e6b8e9b54c9f6baf8bc6ec024692299196bf" Nov 04 01:33:04.182202 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:04.182185 5786 scope.go:122] "RemoveContainer" containerID="f731eeaca63acdac2187872aafc9e6b8e9b54c9f6baf8bc6ec024692299196bf" Nov 04 01:33:04.182444 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:04.182423 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f731eeaca63acdac2187872aafc9e6b8e9b54c9f6baf8bc6ec024692299196bf\": container with ID starting with f731eeaca63acdac2187872aafc9e6b8e9b54c9f6baf8bc6ec024692299196bf not found: ID does not exist" containerID="f731eeaca63acdac2187872aafc9e6b8e9b54c9f6baf8bc6ec024692299196bf" Nov 04 01:33:04.182501 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:04.182452 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f731eeaca63acdac2187872aafc9e6b8e9b54c9f6baf8bc6ec024692299196bf"} err="failed to get container status \"f731eeaca63acdac2187872aafc9e6b8e9b54c9f6baf8bc6ec024692299196bf\": rpc error: code = NotFound desc = could not find container \"f731eeaca63acdac2187872aafc9e6b8e9b54c9f6baf8bc6ec024692299196bf\": container with ID starting with f731eeaca63acdac2187872aafc9e6b8e9b54c9f6baf8bc6ec024692299196bf not found: ID does not exist" Nov 04 01:33:04.230663 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:04.230621 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-7b26ca\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="8ece2c47-d09b-4355-8b9d-7df3a80de475" pod="opendatahub/test-curl-7240-7b26ca" Nov 04 01:33:04.505480 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:04.505410 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ece2c47-d09b-4355-8b9d-7df3a80de475" path="/var/lib/kubelet/pods/8ece2c47-d09b-4355-8b9d-7df3a80de475/volumes" Nov 04 01:33:04.549722 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:04.549678 5786 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-7b26ca\" is forbidden: User \"system:node:ip-10-0-136-94.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-94.us-west-2.compute.internal' and this object" podUID="8ece2c47-d09b-4355-8b9d-7df3a80de475" pod="opendatahub/test-curl-7240-7b26ca" Nov 04 01:33:05.016390 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.016362 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:33:05.030719 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.030691 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:33:05.056627 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.056604 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:33:05.075374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.075344 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" podUID="e496528c-9914-4f1f-bb29-f9d10ac5b3b2" containerName="payload-processing" containerID="cri-o://3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1" gracePeriod=25 Nov 04 01:33:05.138880 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.138845 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-envoy\") pod \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " Nov 04 01:33:05.139048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.138890 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-podinfo\") pod \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " Nov 04 01:33:05.139048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.138910 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-data\") pod \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " Nov 04 01:33:05.139048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.138932 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-kube-api-access-5jx6g\" (UniqueName: \"kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-kube-api-access-5jx6g\") pod \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " Nov 04 01:33:05.139048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.138996 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-data\") pod \"e23550ae-bf0a-44d3-960f-1e10c36b518c\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " Nov 04 01:33:05.139048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139013 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23550ae-bf0a-44d3-960f-1e10c36b518c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23550ae-bf0a-44d3-960f-1e10c36b518c-istiod-ca-cert\") pod \"e23550ae-bf0a-44d3-960f-1e10c36b518c\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " Nov 04 01:33:05.139048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139045 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istiod-ca-cert\") pod \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " Nov 04 01:33:05.139421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139083 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-socket\") pod \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " Nov 04 01:33:05.139421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139143 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-credential-socket\") pod \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " Nov 04 01:33:05.139421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139178 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-token\" (UniqueName: \"kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-token\") pod \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " Nov 04 01:33:05.139421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139204 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-kube-api-access-4m6fr\" (UniqueName: \"kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-kube-api-access-4m6fr\") pod \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " Nov 04 01:33:05.139421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139232 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-podinfo\") pod \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " Nov 04 01:33:05.139421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139228 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-data" pod "dc18916f-4fd9-4086-a1d5-e8036b0d3165" (UID: "dc18916f-4fd9-4086-a1d5-e8036b0d3165"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.139421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139276 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-socket\") pod \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " Nov 04 01:33:05.139421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139317 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-token\" (UniqueName: \"kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-token\") pod \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " Nov 04 01:33:05.139421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139354 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istiod-ca-cert\") pod \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " Nov 04 01:33:05.139421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139395 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-kube-api-access-xvdxz\" (UniqueName: \"kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-kube-api-access-xvdxz\") pod \"e23550ae-bf0a-44d3-960f-1e10c36b518c\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139439 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-credential-socket" pod "a09a5559-a4e6-41f4-9dca-1806e1f61a8e" (UID: "a09a5559-a4e6-41f4-9dca-1806e1f61a8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139470 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-certs\") pod \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139509 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-token\" (UniqueName: \"kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-token\") pod \"e23550ae-bf0a-44d3-960f-1e10c36b518c\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139524 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-socket" pod "dc18916f-4fd9-4086-a1d5-e8036b0d3165" (UID: "dc18916f-4fd9-4086-a1d5-e8036b0d3165"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139457 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istiod-ca-cert" pod "a09a5559-a4e6-41f4-9dca-1806e1f61a8e" (UID: "a09a5559-a4e6-41f4-9dca-1806e1f61a8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139546 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-envoy\") pod \"e23550ae-bf0a-44d3-960f-1e10c36b518c\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139577 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-credential-socket\") pod \"e23550ae-bf0a-44d3-960f-1e10c36b518c\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139615 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-certs\") pod \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139711 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-socket\") pod \"e23550ae-bf0a-44d3-960f-1e10c36b518c\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139755 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-credential-socket\") pod \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\" (UID: \"dc18916f-4fd9-4086-a1d5-e8036b0d3165\") " Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139760 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-data" pod "e23550ae-bf0a-44d3-960f-1e10c36b518c" (UID: "e23550ae-bf0a-44d3-960f-1e10c36b518c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139783 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-podinfo\") pod \"e23550ae-bf0a-44d3-960f-1e10c36b518c\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139813 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-certs\") pod \"e23550ae-bf0a-44d3-960f-1e10c36b518c\" (UID: \"e23550ae-bf0a-44d3-960f-1e10c36b518c\") " Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139848 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-envoy\") pod \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139876 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23550ae-bf0a-44d3-960f-1e10c36b518c-istiod-ca-cert" pod "e23550ae-bf0a-44d3-960f-1e10c36b518c" (UID: "e23550ae-bf0a-44d3-960f-1e10c36b518c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:05.139948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.139906 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-data\") pod \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\" (UID: \"a09a5559-a4e6-41f4-9dca-1806e1f61a8e\") " Nov 04 01:33:05.140807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.140433 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-socket" pod "a09a5559-a4e6-41f4-9dca-1806e1f61a8e" (UID: "a09a5559-a4e6-41f4-9dca-1806e1f61a8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.140807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.140520 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-certs" pod "a09a5559-a4e6-41f4-9dca-1806e1f61a8e" (UID: "a09a5559-a4e6-41f4-9dca-1806e1f61a8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.140807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.140701 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-socket" pod "e23550ae-bf0a-44d3-960f-1e10c36b518c" (UID: "e23550ae-bf0a-44d3-960f-1e10c36b518c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.140973 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.140929 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-certs" pod "e23550ae-bf0a-44d3-960f-1e10c36b518c" (UID: "e23550ae-bf0a-44d3-960f-1e10c36b518c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.140973 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.140933 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-data" pod "a09a5559-a4e6-41f4-9dca-1806e1f61a8e" (UID: "a09a5559-a4e6-41f4-9dca-1806e1f61a8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.141076 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.141016 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-certs" pod "dc18916f-4fd9-4086-a1d5-e8036b0d3165" (UID: "dc18916f-4fd9-4086-a1d5-e8036b0d3165"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.141609 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.141336 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istiod-ca-cert" pod "dc18916f-4fd9-4086-a1d5-e8036b0d3165" (UID: "dc18916f-4fd9-4086-a1d5-e8036b0d3165"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:05.141718 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.141638 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-envoy" pod "dc18916f-4fd9-4086-a1d5-e8036b0d3165" (UID: "dc18916f-4fd9-4086-a1d5-e8036b0d3165"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.141786 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.141749 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-token" pod "dc18916f-4fd9-4086-a1d5-e8036b0d3165" (UID: "dc18916f-4fd9-4086-a1d5-e8036b0d3165"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:05.141949 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.141929 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142051 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142006 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-credential-socket" pod "dc18916f-4fd9-4086-a1d5-e8036b0d3165" (UID: "dc18916f-4fd9-4086-a1d5-e8036b0d3165"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.142200 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142179 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-credential-socket" pod "e23550ae-bf0a-44d3-960f-1e10c36b518c" (UID: "e23550ae-bf0a-44d3-960f-1e10c36b518c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.142263 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142234 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142263 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142256 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142270 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142288 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142302 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142316 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142329 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142342 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142355 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23550ae-bf0a-44d3-960f-1e10c36b518c-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142373 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142386 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142415 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142400 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.142893 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142802 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-kube-api-access-5jx6g" pod "a09a5559-a4e6-41f4-9dca-1806e1f61a8e" (UID: "a09a5559-a4e6-41f4-9dca-1806e1f61a8e"). InnerVolumeSpecName "kube-api-access-5jx6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:05.142893 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142806 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-podinfo" pod "a09a5559-a4e6-41f4-9dca-1806e1f61a8e" (UID: "a09a5559-a4e6-41f4-9dca-1806e1f61a8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:05.143001 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.142957 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-podinfo" pod "dc18916f-4fd9-4086-a1d5-e8036b0d3165" (UID: "dc18916f-4fd9-4086-a1d5-e8036b0d3165"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:05.143430 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.143386 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-podinfo" pod "e23550ae-bf0a-44d3-960f-1e10c36b518c" (UID: "e23550ae-bf0a-44d3-960f-1e10c36b518c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:05.143540 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.143516 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-token" pod "a09a5559-a4e6-41f4-9dca-1806e1f61a8e" (UID: "a09a5559-a4e6-41f4-9dca-1806e1f61a8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:05.143879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.143856 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-envoy" pod "e23550ae-bf0a-44d3-960f-1e10c36b518c" (UID: "e23550ae-bf0a-44d3-960f-1e10c36b518c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.144072 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.144054 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-token" pod "e23550ae-bf0a-44d3-960f-1e10c36b518c" (UID: "e23550ae-bf0a-44d3-960f-1e10c36b518c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:05.144393 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.144376 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-kube-api-access-4m6fr" pod "dc18916f-4fd9-4086-a1d5-e8036b0d3165" (UID: "dc18916f-4fd9-4086-a1d5-e8036b0d3165"). InnerVolumeSpecName "kube-api-access-4m6fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:05.144581 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.144563 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-envoy" pod "a09a5559-a4e6-41f4-9dca-1806e1f61a8e" (UID: "a09a5559-a4e6-41f4-9dca-1806e1f61a8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:05.144646 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.144589 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-kube-api-access-xvdxz" pod "e23550ae-bf0a-44d3-960f-1e10c36b518c" (UID: "e23550ae-bf0a-44d3-960f-1e10c36b518c"). InnerVolumeSpecName "kube-api-access-xvdxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:05.184010 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.183971 5786 generic.go:356] "Generic (PLEG): container finished" podID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerID="c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098" exitCode=0 Nov 04 01:33:05.184240 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.184080 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" event={"ID":"dc18916f-4fd9-4086-a1d5-e8036b0d3165","Type":"ContainerDied","Data":"c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098"} Nov 04 01:33:05.184240 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.184152 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" event={"ID":"dc18916f-4fd9-4086-a1d5-e8036b0d3165","Type":"ContainerDied","Data":"f4358f2b76d92cfcdda025c4e0cdebde7c685ac2733fe2d4fd63b3d7e8e6e04c"} Nov 04 01:33:05.184240 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.184198 5786 scope.go:122] "RemoveContainer" containerID="c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098" Nov 04 01:33:05.184434 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.184314 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz" Nov 04 01:33:05.190195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.190091 5786 generic.go:356] "Generic (PLEG): container finished" podID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerID="4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0" exitCode=0 Nov 04 01:33:05.190429 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.190338 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" Nov 04 01:33:05.190704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.190675 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" event={"ID":"e23550ae-bf0a-44d3-960f-1e10c36b518c","Type":"ContainerDied","Data":"4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0"} Nov 04 01:33:05.190792 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.190739 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr" event={"ID":"e23550ae-bf0a-44d3-960f-1e10c36b518c","Type":"ContainerDied","Data":"5c933daea525cff86ccbeff1726ac4bf98dc6e45e389f109cb61c8f9ce0a7321"} Nov 04 01:33:05.195417 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.195390 5786 generic.go:356] "Generic (PLEG): container finished" podID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerID="ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832" exitCode=0 Nov 04 01:33:05.195544 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.195445 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" Nov 04 01:33:05.195777 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.195449 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" event={"ID":"a09a5559-a4e6-41f4-9dca-1806e1f61a8e","Type":"ContainerDied","Data":"ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832"} Nov 04 01:33:05.195777 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.195738 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2" event={"ID":"a09a5559-a4e6-41f4-9dca-1806e1f61a8e","Type":"ContainerDied","Data":"a964388191b20c96834d845b41f7f4e2e0d581ac280bb79c86bbc62fbaf655fa"} Nov 04 01:33:05.196557 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.196461 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" podUID="978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac" containerName="payload-pre-processing" containerID="cri-o://8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1" gracePeriod=25 Nov 04 01:33:05.203426 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.203314 5786 scope.go:122] "RemoveContainer" containerID="c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098" Nov 04 01:33:05.203720 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:05.203679 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098\": container with ID starting with c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098 not found: ID does not exist" containerID="c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098" Nov 04 01:33:05.203813 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.203717 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098"} err="failed to get container status \"c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098\": rpc error: code = NotFound desc = could not find container \"c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098\": container with ID starting with c9be2c4668bcd939e35ad88098f9de64d15b27309aca67b6533a0b5869dd8098 not found: ID does not exist" Nov 04 01:33:05.203813 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.203740 5786 scope.go:122] "RemoveContainer" containerID="4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0" Nov 04 01:33:05.244377 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244335 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvdxz\" (UniqueName: \"kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-kube-api-access-xvdxz\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244377 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244368 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244385 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244401 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23550ae-bf0a-44d3-960f-1e10c36b518c-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244415 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244429 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23550ae-bf0a-44d3-960f-1e10c36b518c-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244442 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244455 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244469 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244484 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jx6g\" (UniqueName: \"kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-kube-api-access-5jx6g\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244500 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244514 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m6fr\" (UniqueName: \"kubernetes.io/projected/dc18916f-4fd9-4086-a1d5-e8036b0d3165-kube-api-access-4m6fr\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244528 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.244673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.244541 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a5559-a4e6-41f4-9dca-1806e1f61a8e-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.263245 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.263113 5786 scope.go:122] "RemoveContainer" containerID="4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0" Nov 04 01:33:05.263577 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:05.263516 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0\": container with ID starting with 4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0 not found: ID does not exist" containerID="4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0" Nov 04 01:33:05.263878 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.263767 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0"} err="failed to get container status \"4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0\": rpc error: code = NotFound desc = could not find container \"4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0\": container with ID starting with 4be5615b1dab60405c1d22ec80e018d6d654cadd09fd58aa9c251a3ed8c202e0 not found: ID does not exist" Nov 04 01:33:05.263878 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.263819 5786 scope.go:122] "RemoveContainer" containerID="ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832" Nov 04 01:33:05.313521 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.313490 5786 scope.go:122] "RemoveContainer" containerID="97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d" Nov 04 01:33:05.330265 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.330238 5786 scope.go:122] "RemoveContainer" containerID="ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832" Nov 04 01:33:05.330588 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:05.330564 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832\": container with ID starting with ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832 not found: ID does not exist" containerID="ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832" Nov 04 01:33:05.330693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.330600 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832"} err="failed to get container status \"ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832\": rpc error: code = NotFound desc = could not find container \"ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832\": container with ID starting with ab79f10211a24f1dc25a31a787688c866cff9c57770c2694317cbaedd9a48832 not found: ID does not exist" Nov 04 01:33:05.330693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.330625 5786 scope.go:122] "RemoveContainer" containerID="97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d" Nov 04 01:33:05.330919 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:05.330899 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d\": container with ID starting with 97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d not found: ID does not exist" containerID="97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d" Nov 04 01:33:05.330987 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.330929 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d"} err="failed to get container status \"97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d\": rpc error: code = NotFound desc = could not find container \"97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d\": container with ID starting with 97201b94c216d9dc0bf9f667604ae12299c774b819cce2075d41eec544a3347d not found: ID does not exist" Nov 04 01:33:05.352157 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.352130 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" Nov 04 01:33:05.446826 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.446789 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-plugins-config-volume\") pod \"e496528c-9914-4f1f-bb29-f9d10ac5b3b2\" (UID: \"e496528c-9914-4f1f-bb29-f9d10ac5b3b2\") " Nov 04 01:33:05.447026 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.446989 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-kube-api-access-fbbsj\" (UniqueName: \"kubernetes.io/projected/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-kube-api-access-fbbsj\") pod \"e496528c-9914-4f1f-bb29-f9d10ac5b3b2\" (UID: \"e496528c-9914-4f1f-bb29-f9d10ac5b3b2\") " Nov 04 01:33:05.447291 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.447236 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-plugins-config-volume" pod "e496528c-9914-4f1f-bb29-f9d10ac5b3b2" (UID: "e496528c-9914-4f1f-bb29-f9d10ac5b3b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:05.447476 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.447458 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.450276 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.450248 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-kube-api-access-fbbsj" pod "e496528c-9914-4f1f-bb29-f9d10ac5b3b2" (UID: "e496528c-9914-4f1f-bb29-f9d10ac5b3b2"). InnerVolumeSpecName "kube-api-access-fbbsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:05.471360 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.471338 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" Nov 04 01:33:05.529750 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.529720 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr"] Nov 04 01:33:05.534069 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.534040 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-mk2tr"] Nov 04 01:33:05.548227 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.548198 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-kube-api-access-mnkwn\" (UniqueName: \"kubernetes.io/projected/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-kube-api-access-mnkwn\") pod \"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac\" (UID: \"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac\") " Nov 04 01:33:05.548387 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.548329 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-plugins-config-volume\") pod \"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac\" (UID: \"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac\") " Nov 04 01:33:05.548673 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.548657 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbbsj\" (UniqueName: \"kubernetes.io/projected/e496528c-9914-4f1f-bb29-f9d10ac5b3b2-kube-api-access-fbbsj\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.548733 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.548702 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-plugins-config-volume" pod "978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac" (UID: "978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:05.550365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.550345 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-kube-api-access-mnkwn" pod "978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac" (UID: "978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac"). InnerVolumeSpecName "kube-api-access-mnkwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:05.649911 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.649883 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnkwn\" (UniqueName: \"kubernetes.io/projected/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-kube-api-access-mnkwn\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.649911 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.649910 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.747892 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.747856 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2"] Nov 04 01:33:05.755648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.755617 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-p9mp2"] Nov 04 01:33:05.864077 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.864041 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz"] Nov 04 01:33:05.866940 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:05.866910 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-qplkz"] Nov 04 01:33:06.207956 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.207922 5786 generic.go:356] "Generic (PLEG): container finished" podID="e496528c-9914-4f1f-bb29-f9d10ac5b3b2" containerID="3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1" exitCode=0 Nov 04 01:33:06.208166 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.207973 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" Nov 04 01:33:06.208166 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.208007 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" event={"ID":"e496528c-9914-4f1f-bb29-f9d10ac5b3b2","Type":"ContainerDied","Data":"3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1"} Nov 04 01:33:06.208166 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.208063 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv" event={"ID":"e496528c-9914-4f1f-bb29-f9d10ac5b3b2","Type":"ContainerDied","Data":"52d3120bcf8fca615dde9cc31fbe3c3bb4f16a0b70662171ab2c0645610e0a02"} Nov 04 01:33:06.208166 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.208092 5786 scope.go:122] "RemoveContainer" containerID="3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1" Nov 04 01:33:06.211655 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.211628 5786 generic.go:356] "Generic (PLEG): container finished" podID="978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac" containerID="8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1" exitCode=0 Nov 04 01:33:06.211793 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.211676 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" Nov 04 01:33:06.211793 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.211725 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" event={"ID":"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac","Type":"ContainerDied","Data":"8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1"} Nov 04 01:33:06.211793 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.211762 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d" event={"ID":"978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac","Type":"ContainerDied","Data":"bb14843f8c1efad9f1a0dfd99f0bf38a542108caff5c3c5c9b3fa76c19fd576a"} Nov 04 01:33:06.218601 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.218585 5786 scope.go:122] "RemoveContainer" containerID="3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1" Nov 04 01:33:06.218887 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:06.218861 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1\": container with ID starting with 3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1 not found: ID does not exist" containerID="3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1" Nov 04 01:33:06.218946 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.218901 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1"} err="failed to get container status \"3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1\": rpc error: code = NotFound desc = could not find container \"3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1\": container with ID starting with 3cba3aede1d4258028002eb96294f45588ed24e5333f24c30b5d21b6667bdfb1 not found: ID does not exist" Nov 04 01:33:06.218946 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.218918 5786 scope.go:122] "RemoveContainer" containerID="8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1" Nov 04 01:33:06.227828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.227813 5786 scope.go:122] "RemoveContainer" containerID="8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1" Nov 04 01:33:06.228079 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:06.228061 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1\": container with ID starting with 8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1 not found: ID does not exist" containerID="8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1" Nov 04 01:33:06.228138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.228085 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1"} err="failed to get container status \"8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1\": rpc error: code = NotFound desc = could not find container \"8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1\": container with ID starting with 8c365bfbcf58d71933b0682f60178853f1024b66d7c8250087df7f83e989fae1 not found: ID does not exist" Nov 04 01:33:06.371929 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.371894 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d"] Nov 04 01:33:06.376648 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.376623 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-fc46df-6d4ff657f-ddw7d"] Nov 04 01:33:06.482198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.482099 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv"] Nov 04 01:33:06.487614 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.487585 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-fc46df-6f56795566-fj8pv"] Nov 04 01:33:06.506022 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.505993 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac" path="/var/lib/kubelet/pods/978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac/volumes" Nov 04 01:33:06.506692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.506668 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" path="/var/lib/kubelet/pods/a09a5559-a4e6-41f4-9dca-1806e1f61a8e/volumes" Nov 04 01:33:06.507293 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.507275 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" path="/var/lib/kubelet/pods/dc18916f-4fd9-4086-a1d5-e8036b0d3165/volumes" Nov 04 01:33:06.508215 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.508195 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" path="/var/lib/kubelet/pods/e23550ae-bf0a-44d3-960f-1e10c36b518c/volumes" Nov 04 01:33:06.508664 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:06.508647 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e496528c-9914-4f1f-bb29-f9d10ac5b3b2" path="/var/lib/kubelet/pods/e496528c-9914-4f1f-bb29-f9d10ac5b3b2/volumes" Nov 04 01:33:13.060824 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:13.060788 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c"] Nov 04 01:33:13.179481 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:13.179446 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq"] Nov 04 01:33:15.086901 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:15.086857 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:33:18.061413 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.061374 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" podUID="f10046d1-f25f-4ccd-9139-1f2a864d84e7" containerName="payload-processing" containerID="cri-o://a734d73eb64e42a76216b0f4a80cb4837e7d012301ad4bcc326e4a9515cc919e" gracePeriod=25 Nov 04 01:33:18.180747 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.180714 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" podUID="fcd9752e-6a9b-470f-ad9d-938bca9a46cc" containerName="payload-pre-processing" containerID="cri-o://d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4" gracePeriod=25 Nov 04 01:33:18.297712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.295866 5786 generic.go:356] "Generic (PLEG): container finished" podID="f10046d1-f25f-4ccd-9139-1f2a864d84e7" containerID="a734d73eb64e42a76216b0f4a80cb4837e7d012301ad4bcc326e4a9515cc919e" exitCode=0 Nov 04 01:33:18.297712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.295983 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" event={"ID":"f10046d1-f25f-4ccd-9139-1f2a864d84e7","Type":"ContainerDied","Data":"a734d73eb64e42a76216b0f4a80cb4837e7d012301ad4bcc326e4a9515cc919e"} Nov 04 01:33:18.334516 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.331672 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" Nov 04 01:33:18.433224 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.433198 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10046d1-f25f-4ccd-9139-1f2a864d84e7-kube-api-access-6qqhb\" (UniqueName: \"kubernetes.io/projected/f10046d1-f25f-4ccd-9139-1f2a864d84e7-kube-api-access-6qqhb\") pod \"f10046d1-f25f-4ccd-9139-1f2a864d84e7\" (UID: \"f10046d1-f25f-4ccd-9139-1f2a864d84e7\") " Nov 04 01:33:18.433376 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.433278 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10046d1-f25f-4ccd-9139-1f2a864d84e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10046d1-f25f-4ccd-9139-1f2a864d84e7-plugins-config-volume\") pod \"f10046d1-f25f-4ccd-9139-1f2a864d84e7\" (UID: \"f10046d1-f25f-4ccd-9139-1f2a864d84e7\") " Nov 04 01:33:18.433652 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.433627 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10046d1-f25f-4ccd-9139-1f2a864d84e7-plugins-config-volume" pod "f10046d1-f25f-4ccd-9139-1f2a864d84e7" (UID: "f10046d1-f25f-4ccd-9139-1f2a864d84e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:18.435525 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.435493 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10046d1-f25f-4ccd-9139-1f2a864d84e7-kube-api-access-6qqhb" pod "f10046d1-f25f-4ccd-9139-1f2a864d84e7" (UID: "f10046d1-f25f-4ccd-9139-1f2a864d84e7"). InnerVolumeSpecName "kube-api-access-6qqhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:18.497075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.497037 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" Nov 04 01:33:18.534559 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.534531 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10046d1-f25f-4ccd-9139-1f2a864d84e7-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:18.534559 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.534556 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qqhb\" (UniqueName: \"kubernetes.io/projected/f10046d1-f25f-4ccd-9139-1f2a864d84e7-kube-api-access-6qqhb\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:18.644952 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.644872 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-plugins-config-volume\") pod \"fcd9752e-6a9b-470f-ad9d-938bca9a46cc\" (UID: \"fcd9752e-6a9b-470f-ad9d-938bca9a46cc\") " Nov 04 01:33:18.644952 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.644931 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-kube-api-access-l8464\" (UniqueName: \"kubernetes.io/projected/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-kube-api-access-l8464\") pod \"fcd9752e-6a9b-470f-ad9d-938bca9a46cc\" (UID: \"fcd9752e-6a9b-470f-ad9d-938bca9a46cc\") " Nov 04 01:33:18.646546 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.646224 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-plugins-config-volume" pod "fcd9752e-6a9b-470f-ad9d-938bca9a46cc" (UID: "fcd9752e-6a9b-470f-ad9d-938bca9a46cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:18.662669 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.656278 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-kube-api-access-l8464" pod "fcd9752e-6a9b-470f-ad9d-938bca9a46cc" (UID: "fcd9752e-6a9b-470f-ad9d-938bca9a46cc"). InnerVolumeSpecName "kube-api-access-l8464". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:18.746406 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.746367 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:18.746406 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:18.746400 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8464\" (UniqueName: \"kubernetes.io/projected/fcd9752e-6a9b-470f-ad9d-938bca9a46cc-kube-api-access-l8464\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:19.320136 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.320072 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" event={"ID":"f10046d1-f25f-4ccd-9139-1f2a864d84e7","Type":"ContainerDied","Data":"2d32972cba6f29acbf44c8185b3828bb236b886d9ecd94e10456970bab1602aa"} Nov 04 01:33:19.324731 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.320166 5786 scope.go:122] "RemoveContainer" containerID="a734d73eb64e42a76216b0f4a80cb4837e7d012301ad4bcc326e4a9515cc919e" Nov 04 01:33:19.324731 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.320359 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c" Nov 04 01:33:19.330193 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.330073 5786 generic.go:356] "Generic (PLEG): container finished" podID="fcd9752e-6a9b-470f-ad9d-938bca9a46cc" containerID="d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4" exitCode=0 Nov 04 01:33:19.332490 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.331201 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" Nov 04 01:33:19.333048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.333027 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" event={"ID":"fcd9752e-6a9b-470f-ad9d-938bca9a46cc","Type":"ContainerDied","Data":"d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4"} Nov 04 01:33:19.333278 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.333241 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq" event={"ID":"fcd9752e-6a9b-470f-ad9d-938bca9a46cc","Type":"ContainerDied","Data":"c0d56a3388aa8e52b71f2f2ec01e88c7ea82eb4ca9b79412868513032782c5b7"} Nov 04 01:33:19.354085 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.353288 5786 scope.go:122] "RemoveContainer" containerID="d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4" Nov 04 01:33:19.368332 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.368309 5786 scope.go:122] "RemoveContainer" containerID="d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4" Nov 04 01:33:19.368651 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:19.368630 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4\": container with ID starting with d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4 not found: ID does not exist" containerID="d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4" Nov 04 01:33:19.368719 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.368663 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4"} err="failed to get container status \"d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4\": rpc error: code = NotFound desc = could not find container \"d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4\": container with ID starting with d2e3d9797e906c1d9ffdb85d35f38148c52e1f65b08e24a6bf936345be774ec4 not found: ID does not exist" Nov 04 01:33:19.474424 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.474384 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57"] Nov 04 01:33:19.474598 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.474577 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" podUID="ce0ed274-cc86-4ab4-b8fb-db31ee650267" containerName="maas-api" containerID="cri-o://135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10" gracePeriod=30 Nov 04 01:33:19.486952 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.486924 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c"] Nov 04 01:33:19.523884 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.523848 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-98f846-6b48569ccd-7qg4c"] Nov 04 01:33:19.614473 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.614437 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq"] Nov 04 01:33:19.625858 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.625814 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98f846-68bcf6c695-jcnmq"] Nov 04 01:33:19.740259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.740234 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:33:19.903185 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.903141 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0ed274-cc86-4ab4-b8fb-db31ee650267-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0ed274-cc86-4ab4-b8fb-db31ee650267-openshift-service-ca\") pod \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " Nov 04 01:33:19.903185 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.903190 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-maas-api-tls\") pod \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " Nov 04 01:33:19.903461 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.903212 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0ed274-cc86-4ab4-b8fb-db31ee650267-kube-api-access-tvtdj\" (UniqueName: \"kubernetes.io/projected/ce0ed274-cc86-4ab4-b8fb-db31ee650267-kube-api-access-tvtdj\") pod \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " Nov 04 01:33:19.903461 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.903453 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-metrics-certs\") pod \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\" (UID: \"ce0ed274-cc86-4ab4-b8fb-db31ee650267\") " Nov 04 01:33:19.903583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.903533 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0ed274-cc86-4ab4-b8fb-db31ee650267-openshift-service-ca" pod "ce0ed274-cc86-4ab4-b8fb-db31ee650267" (UID: "ce0ed274-cc86-4ab4-b8fb-db31ee650267"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:19.903841 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.903822 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0ed274-cc86-4ab4-b8fb-db31ee650267-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:19.905440 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.905413 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-maas-api-tls" pod "ce0ed274-cc86-4ab4-b8fb-db31ee650267" (UID: "ce0ed274-cc86-4ab4-b8fb-db31ee650267"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:19.905530 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.905468 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0ed274-cc86-4ab4-b8fb-db31ee650267-kube-api-access-tvtdj" pod "ce0ed274-cc86-4ab4-b8fb-db31ee650267" (UID: "ce0ed274-cc86-4ab4-b8fb-db31ee650267"). InnerVolumeSpecName "kube-api-access-tvtdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:19.905530 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:19.905474 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-metrics-certs" pod "ce0ed274-cc86-4ab4-b8fb-db31ee650267" (UID: "ce0ed274-cc86-4ab4-b8fb-db31ee650267"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:20.005251 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.005218 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:20.005251 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.005250 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvtdj\" (UniqueName: \"kubernetes.io/projected/ce0ed274-cc86-4ab4-b8fb-db31ee650267-kube-api-access-tvtdj\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:20.005448 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.005261 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0ed274-cc86-4ab4-b8fb-db31ee650267-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:20.367537 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.367498 5786 generic.go:356] "Generic (PLEG): container finished" podID="ce0ed274-cc86-4ab4-b8fb-db31ee650267" containerID="135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10" exitCode=0 Nov 04 01:33:20.368058 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.367558 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" event={"ID":"ce0ed274-cc86-4ab4-b8fb-db31ee650267","Type":"ContainerDied","Data":"135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10"} Nov 04 01:33:20.368058 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.367655 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" event={"ID":"ce0ed274-cc86-4ab4-b8fb-db31ee650267","Type":"ContainerDied","Data":"a09ff1b839c45f09ab6e7c3fc673179d39d59722607c160da65aede839e71f53"} Nov 04 01:33:20.368058 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.367681 5786 scope.go:122] "RemoveContainer" containerID="135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10" Nov 04 01:33:20.368058 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.367866 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57" Nov 04 01:33:20.382627 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.382554 5786 scope.go:122] "RemoveContainer" containerID="135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10" Nov 04 01:33:20.383066 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:20.383041 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10\": container with ID starting with 135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10 not found: ID does not exist" containerID="135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10" Nov 04 01:33:20.383214 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.383077 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10"} err="failed to get container status \"135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10\": rpc error: code = NotFound desc = could not find container \"135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10\": container with ID starting with 135a3752f526e35a5eb37c24721cfac95e85c59adcd06884d4d5e79223e78d10 not found: ID does not exist" Nov 04 01:33:20.544141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.539306 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10046d1-f25f-4ccd-9139-1f2a864d84e7" path="/var/lib/kubelet/pods/f10046d1-f25f-4ccd-9139-1f2a864d84e7/volumes" Nov 04 01:33:20.544141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.540003 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd9752e-6a9b-470f-ad9d-938bca9a46cc" path="/var/lib/kubelet/pods/fcd9752e-6a9b-470f-ad9d-938bca9a46cc/volumes" Nov 04 01:33:20.593246 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.593208 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57"] Nov 04 01:33:20.594085 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:20.594056 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-fc46df-65fbd4548c-lqp57"] Nov 04 01:33:22.504735 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:22.504704 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0ed274-cc86-4ab4-b8fb-db31ee650267" path="/var/lib/kubelet/pods/ce0ed274-cc86-4ab4-b8fb-db31ee650267/volumes" Nov 04 01:33:24.996646 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:24.996608 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm"] Nov 04 01:33:24.997197 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:24.996977 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" containerID="cri-o://257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62" gracePeriod=30 Nov 04 01:33:25.111593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:25.111558 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp"] Nov 04 01:33:25.111816 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:25.111779 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" containerID="cri-o://017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e" gracePeriod=30 Nov 04 01:33:27.900130 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:27.900080 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g"] Nov 04 01:33:27.900561 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:27.900363 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" containerID="cri-o://b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79" gracePeriod=30 Nov 04 01:33:29.782039 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:29.782000 5786 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:29.782469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:29.782063 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:29.908561 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:29.908525 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:29.908717 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:29.908599 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:30.265258 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.264745 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:33:30.376642 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.376612 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:33:30.381705 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.381684 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-envoy\") pod \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " Nov 04 01:33:30.381799 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.381728 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istiod-ca-cert\") pod \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " Nov 04 01:33:30.381905 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.381886 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-podinfo\") pod \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " Nov 04 01:33:30.381992 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.381960 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-certs\") pod \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " Nov 04 01:33:30.382048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.382012 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-socket\") pod \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " Nov 04 01:33:30.382110 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.382073 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-kube-api-access-hpbsx\" (UniqueName: \"kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-kube-api-access-hpbsx\") pod \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " Nov 04 01:33:30.382194 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.382137 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-token\" (UniqueName: \"kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-token\") pod \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " Nov 04 01:33:30.382194 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.382185 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-data\") pod \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " Nov 04 01:33:30.382308 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.382200 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istiod-ca-cert" pod "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" (UID: "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:30.382308 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.382232 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-credential-socket\") pod \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\" (UID: \"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2\") " Nov 04 01:33:30.382484 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.382458 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-certs" pod "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" (UID: "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:30.382716 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.382692 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-credential-socket" pod "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" (UID: "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:30.383035 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.382819 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.383035 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.382846 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.383035 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.382860 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.383035 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.382979 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-socket" pod "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" (UID: "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:30.383586 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.383556 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-data" pod "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" (UID: "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:30.384359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.384336 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-podinfo" pod "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" (UID: "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:30.384472 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.384450 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-envoy" pod "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" (UID: "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:30.384751 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.384732 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-kube-api-access-hpbsx" pod "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" (UID: "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2"). InnerVolumeSpecName "kube-api-access-hpbsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:30.384896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.384873 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-token" pod "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" (UID: "c4e0983c-143e-4cc0-a647-8a1ac8d18ff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:30.451176 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.451103 5786 generic.go:356] "Generic (PLEG): container finished" podID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerID="017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e" exitCode=0 Nov 04 01:33:30.451354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.451186 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" Nov 04 01:33:30.451354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.451150 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" event={"ID":"f57ffc7e-b9c6-44e0-a5d3-08df226ee170","Type":"ContainerDied","Data":"017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e"} Nov 04 01:33:30.451354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.451282 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp" event={"ID":"f57ffc7e-b9c6-44e0-a5d3-08df226ee170","Type":"ContainerDied","Data":"3d9494940b6caa42137e84321ac0da9ea2fab631a503973e57326978f63a4149"} Nov 04 01:33:30.451354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.451305 5786 scope.go:122] "RemoveContainer" containerID="017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e" Nov 04 01:33:30.453109 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.452964 5786 generic.go:356] "Generic (PLEG): container finished" podID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerID="257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62" exitCode=0 Nov 04 01:33:30.453109 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.452993 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" event={"ID":"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2","Type":"ContainerDied","Data":"257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62"} Nov 04 01:33:30.453109 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.453020 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" Nov 04 01:33:30.453109 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.453037 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm" event={"ID":"c4e0983c-143e-4cc0-a647-8a1ac8d18ff2","Type":"ContainerDied","Data":"67ba53d2404ce5bb08ae4fdcaa4f3295d5b53a58504402288113fe7a8c7dc50c"} Nov 04 01:33:30.461388 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.461109 5786 scope.go:122] "RemoveContainer" containerID="017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e" Nov 04 01:33:30.461630 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:30.461608 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e\": container with ID starting with 017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e not found: ID does not exist" containerID="017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e" Nov 04 01:33:30.461746 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.461640 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e"} err="failed to get container status \"017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e\": rpc error: code = NotFound desc = could not find container \"017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e\": container with ID starting with 017ea1639e239a793e62340de82c343b80b4202ba0de82a5e2b54c982b354c9e not found: ID does not exist" Nov 04 01:33:30.461746 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.461663 5786 scope.go:122] "RemoveContainer" containerID="257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62" Nov 04 01:33:30.471393 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.471371 5786 scope.go:122] "RemoveContainer" containerID="257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62" Nov 04 01:33:30.471708 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:30.471683 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62\": container with ID starting with 257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62 not found: ID does not exist" containerID="257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62" Nov 04 01:33:30.471774 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.471720 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62"} err="failed to get container status \"257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62\": rpc error: code = NotFound desc = could not find container \"257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62\": container with ID starting with 257f835981530462d4bdfaee821e7cd58e4e266ab44f1198db1776b483294e62 not found: ID does not exist" Nov 04 01:33:30.484149 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484105 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-podinfo\") pod \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " Nov 04 01:33:30.484272 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484261 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-data\") pod \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " Nov 04 01:33:30.484330 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484305 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-socket\") pod \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " Nov 04 01:33:30.484365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484343 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-kube-api-access-kbzgj\" (UniqueName: \"kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-kube-api-access-kbzgj\") pod \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " Nov 04 01:33:30.484444 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484423 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-credential-socket\") pod \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " Nov 04 01:33:30.484602 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484489 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istiod-ca-cert\") pod \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " Nov 04 01:33:30.484602 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484528 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-token\" (UniqueName: \"kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-token\") pod \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " Nov 04 01:33:30.484602 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484574 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-certs\") pod \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " Nov 04 01:33:30.484602 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484563 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-data" pod "f57ffc7e-b9c6-44e0-a5d3-08df226ee170" (UID: "f57ffc7e-b9c6-44e0-a5d3-08df226ee170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:30.484846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484621 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-envoy\") pod \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\" (UID: \"f57ffc7e-b9c6-44e0-a5d3-08df226ee170\") " Nov 04 01:33:30.484846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484623 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-socket" pod "f57ffc7e-b9c6-44e0-a5d3-08df226ee170" (UID: "f57ffc7e-b9c6-44e0-a5d3-08df226ee170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:30.484846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484641 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-credential-socket" pod "f57ffc7e-b9c6-44e0-a5d3-08df226ee170" (UID: "f57ffc7e-b9c6-44e0-a5d3-08df226ee170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:30.484846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484812 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-certs" pod "f57ffc7e-b9c6-44e0-a5d3-08df226ee170" (UID: "f57ffc7e-b9c6-44e0-a5d3-08df226ee170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:30.485067 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.484895 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istiod-ca-cert" pod "f57ffc7e-b9c6-44e0-a5d3-08df226ee170" (UID: "f57ffc7e-b9c6-44e0-a5d3-08df226ee170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:30.485603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.485422 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.485603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.485452 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.485603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.485466 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.485603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.485482 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpbsx\" (UniqueName: \"kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-kube-api-access-hpbsx\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.485603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.485495 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.485603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.485508 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.485603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.485520 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.485603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.485534 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.485603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.485548 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.485603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.485560 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.485603 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.485573 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.488303 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.488253 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-podinfo" pod "f57ffc7e-b9c6-44e0-a5d3-08df226ee170" (UID: "f57ffc7e-b9c6-44e0-a5d3-08df226ee170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:30.488303 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.488254 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-envoy" pod "f57ffc7e-b9c6-44e0-a5d3-08df226ee170" (UID: "f57ffc7e-b9c6-44e0-a5d3-08df226ee170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:30.488484 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.488302 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-token" pod "f57ffc7e-b9c6-44e0-a5d3-08df226ee170" (UID: "f57ffc7e-b9c6-44e0-a5d3-08df226ee170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:30.488484 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.488334 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-kube-api-access-kbzgj" pod "f57ffc7e-b9c6-44e0-a5d3-08df226ee170" (UID: "f57ffc7e-b9c6-44e0-a5d3-08df226ee170"). InnerVolumeSpecName "kube-api-access-kbzgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:30.586437 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.586349 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbzgj\" (UniqueName: \"kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-kube-api-access-kbzgj\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.586437 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.586377 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.586437 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.586387 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.586437 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.586395 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57ffc7e-b9c6-44e0-a5d3-08df226ee170-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.716710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.716677 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm"] Nov 04 01:33:30.729849 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.729808 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-cqdnm"] Nov 04 01:33:30.861531 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.861494 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp"] Nov 04 01:33:30.866209 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:30.866181 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-dglrp"] Nov 04 01:33:32.512935 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:32.512879 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" path="/var/lib/kubelet/pods/c4e0983c-143e-4cc0-a647-8a1ac8d18ff2/volumes" Nov 04 01:33:32.514148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:32.514107 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" path="/var/lib/kubelet/pods/f57ffc7e-b9c6-44e0-a5d3-08df226ee170/volumes" Nov 04 01:33:33.155256 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.155227 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:33:33.212602 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.212520 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-kube-api-access-l6rnk\" (UniqueName: \"kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-kube-api-access-l6rnk\") pod \"febf58ae-2630-40f9-a463-be0d0c583f2b\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " Nov 04 01:33:33.212602 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.212565 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-podinfo\") pod \"febf58ae-2630-40f9-a463-be0d0c583f2b\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " Nov 04 01:33:33.212602 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.212593 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-token\" (UniqueName: \"kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-token\") pod \"febf58ae-2630-40f9-a463-be0d0c583f2b\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " Nov 04 01:33:33.212883 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.212637 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-socket\") pod \"febf58ae-2630-40f9-a463-be0d0c583f2b\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " Nov 04 01:33:33.212883 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.212662 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-certs\") pod \"febf58ae-2630-40f9-a463-be0d0c583f2b\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " Nov 04 01:33:33.212883 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.212748 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-credential-socket\") pod \"febf58ae-2630-40f9-a463-be0d0c583f2b\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " Nov 04 01:33:33.212883 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.212806 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-data\") pod \"febf58ae-2630-40f9-a463-be0d0c583f2b\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " Nov 04 01:33:33.212883 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.212835 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-envoy\") pod \"febf58ae-2630-40f9-a463-be0d0c583f2b\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " Nov 04 01:33:33.213169 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.212928 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/febf58ae-2630-40f9-a463-be0d0c583f2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febf58ae-2630-40f9-a463-be0d0c583f2b-istiod-ca-cert\") pod \"febf58ae-2630-40f9-a463-be0d0c583f2b\" (UID: \"febf58ae-2630-40f9-a463-be0d0c583f2b\") " Nov 04 01:33:33.213169 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.212999 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-certs" pod "febf58ae-2630-40f9-a463-be0d0c583f2b" (UID: "febf58ae-2630-40f9-a463-be0d0c583f2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.213169 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.213012 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-socket" pod "febf58ae-2630-40f9-a463-be0d0c583f2b" (UID: "febf58ae-2630-40f9-a463-be0d0c583f2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.213339 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.213180 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-data" pod "febf58ae-2630-40f9-a463-be0d0c583f2b" (UID: "febf58ae-2630-40f9-a463-be0d0c583f2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.213339 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.213308 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-credential-socket" pod "febf58ae-2630-40f9-a463-be0d0c583f2b" (UID: "febf58ae-2630-40f9-a463-be0d0c583f2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.213442 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.213422 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/febf58ae-2630-40f9-a463-be0d0c583f2b-istiod-ca-cert" pod "febf58ae-2630-40f9-a463-be0d0c583f2b" (UID: "febf58ae-2630-40f9-a463-be0d0c583f2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:33.213504 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.213480 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.213504 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.213499 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.213615 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.213508 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.213615 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.213516 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.215000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.214979 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-kube-api-access-l6rnk" pod "febf58ae-2630-40f9-a463-be0d0c583f2b" (UID: "febf58ae-2630-40f9-a463-be0d0c583f2b"). InnerVolumeSpecName "kube-api-access-l6rnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:33.215000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.214976 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-envoy" pod "febf58ae-2630-40f9-a463-be0d0c583f2b" (UID: "febf58ae-2630-40f9-a463-be0d0c583f2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.215260 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.215241 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-token" pod "febf58ae-2630-40f9-a463-be0d0c583f2b" (UID: "febf58ae-2630-40f9-a463-be0d0c583f2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:33.215260 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.215241 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-podinfo" pod "febf58ae-2630-40f9-a463-be0d0c583f2b" (UID: "febf58ae-2630-40f9-a463-be0d0c583f2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:33.314167 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.314134 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.314167 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.314168 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febf58ae-2630-40f9-a463-be0d0c583f2b-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.314358 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.314179 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6rnk\" (UniqueName: \"kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-kube-api-access-l6rnk\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.314358 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.314190 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.314358 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.314201 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/febf58ae-2630-40f9-a463-be0d0c583f2b-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.480086 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.479994 5786 generic.go:356] "Generic (PLEG): container finished" podID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerID="b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79" exitCode=0 Nov 04 01:33:33.480270 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.480085 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" event={"ID":"febf58ae-2630-40f9-a463-be0d0c583f2b","Type":"ContainerDied","Data":"b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79"} Nov 04 01:33:33.480270 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.480171 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" Nov 04 01:33:33.480270 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.480190 5786 scope.go:122] "RemoveContainer" containerID="b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79" Nov 04 01:33:33.480270 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.480177 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g" event={"ID":"febf58ae-2630-40f9-a463-be0d0c583f2b","Type":"ContainerDied","Data":"e7971be82f38123c9e9aaceefc4c597df4dd898dd89ac4620b6953d96856d496"} Nov 04 01:33:33.490713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.490689 5786 scope.go:122] "RemoveContainer" containerID="d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed" Nov 04 01:33:33.499322 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.499249 5786 scope.go:122] "RemoveContainer" containerID="b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79" Nov 04 01:33:33.499520 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:33.499498 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79\": container with ID starting with b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79 not found: ID does not exist" containerID="b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79" Nov 04 01:33:33.499569 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.499527 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79"} err="failed to get container status \"b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79\": rpc error: code = NotFound desc = could not find container \"b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79\": container with ID starting with b0b8b6907bc045ddb14ac41f148dfbc129f876287bc6ae3c20f990c14de91a79 not found: ID does not exist" Nov 04 01:33:33.499569 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.499544 5786 scope.go:122] "RemoveContainer" containerID="d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed" Nov 04 01:33:33.499764 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:33.499748 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed\": container with ID starting with d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed not found: ID does not exist" containerID="d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed" Nov 04 01:33:33.499815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.499766 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed"} err="failed to get container status \"d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed\": rpc error: code = NotFound desc = could not find container \"d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed\": container with ID starting with d294dbede6daa9da1fcc0dbcd62f1e1304c4bb656fac1993a82f50eabca055ed not found: ID does not exist" Nov 04 01:33:33.640138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.640091 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g"] Nov 04 01:33:33.648018 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:33.647990 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-7qj5g"] Nov 04 01:33:34.504996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:34.504963 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" path="/var/lib/kubelet/pods/febf58ae-2630-40f9-a463-be0d0c583f2b/volumes" Nov 04 01:33:35.441807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:35.441774 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck"] Nov 04 01:33:36.970327 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:36.970292 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j"] Nov 04 01:33:36.970791 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:36.970503 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" podUID="08f1f1d7-b68a-489c-8914-6525a4079c6a" containerName="maas-api" containerID="cri-o://9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd" gracePeriod=30 Nov 04 01:33:37.255476 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.255453 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:33:37.359593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.359555 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f1f1d7-b68a-489c-8914-6525a4079c6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f1f1d7-b68a-489c-8914-6525a4079c6a-openshift-service-ca\") pod \"08f1f1d7-b68a-489c-8914-6525a4079c6a\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " Nov 04 01:33:37.359593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.359596 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f1f1d7-b68a-489c-8914-6525a4079c6a-kube-api-access-s5mrh\" (UniqueName: \"kubernetes.io/projected/08f1f1d7-b68a-489c-8914-6525a4079c6a-kube-api-access-s5mrh\") pod \"08f1f1d7-b68a-489c-8914-6525a4079c6a\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " Nov 04 01:33:37.359866 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.359629 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-metrics-certs\") pod \"08f1f1d7-b68a-489c-8914-6525a4079c6a\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " Nov 04 01:33:37.359866 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.359710 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-maas-api-tls\") pod \"08f1f1d7-b68a-489c-8914-6525a4079c6a\" (UID: \"08f1f1d7-b68a-489c-8914-6525a4079c6a\") " Nov 04 01:33:37.360031 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.360002 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f1f1d7-b68a-489c-8914-6525a4079c6a-openshift-service-ca" pod "08f1f1d7-b68a-489c-8914-6525a4079c6a" (UID: "08f1f1d7-b68a-489c-8914-6525a4079c6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:37.361838 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.361808 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-maas-api-tls" pod "08f1f1d7-b68a-489c-8914-6525a4079c6a" (UID: "08f1f1d7-b68a-489c-8914-6525a4079c6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:37.361838 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.361820 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-metrics-certs" pod "08f1f1d7-b68a-489c-8914-6525a4079c6a" (UID: "08f1f1d7-b68a-489c-8914-6525a4079c6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:37.361991 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.361925 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f1f1d7-b68a-489c-8914-6525a4079c6a-kube-api-access-s5mrh" pod "08f1f1d7-b68a-489c-8914-6525a4079c6a" (UID: "08f1f1d7-b68a-489c-8914-6525a4079c6a"). InnerVolumeSpecName "kube-api-access-s5mrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:37.469417 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.469373 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f1f1d7-b68a-489c-8914-6525a4079c6a-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:37.469631 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.469432 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5mrh\" (UniqueName: \"kubernetes.io/projected/08f1f1d7-b68a-489c-8914-6525a4079c6a-kube-api-access-s5mrh\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:37.469708 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.469650 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:37.469708 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.469670 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f1f1d7-b68a-489c-8914-6525a4079c6a-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:37.502353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.502267 5786 generic.go:356] "Generic (PLEG): container finished" podID="08f1f1d7-b68a-489c-8914-6525a4079c6a" containerID="9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd" exitCode=0 Nov 04 01:33:37.502353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.502325 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" Nov 04 01:33:37.502553 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.502347 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" event={"ID":"08f1f1d7-b68a-489c-8914-6525a4079c6a","Type":"ContainerDied","Data":"9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd"} Nov 04 01:33:37.502553 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.502400 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j" event={"ID":"08f1f1d7-b68a-489c-8914-6525a4079c6a","Type":"ContainerDied","Data":"26114c987eaa47f1a7c51e1dab239d6183680b8cb944c5b2029e7c3fefeeb1c4"} Nov 04 01:33:37.502553 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.502428 5786 scope.go:122] "RemoveContainer" containerID="9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd" Nov 04 01:33:37.513905 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.513882 5786 scope.go:122] "RemoveContainer" containerID="9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd" Nov 04 01:33:37.514242 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:37.514220 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd\": container with ID starting with 9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd not found: ID does not exist" containerID="9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd" Nov 04 01:33:37.514294 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.514254 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd"} err="failed to get container status \"9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd\": rpc error: code = NotFound desc = could not find container \"9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd\": container with ID starting with 9be876d523869badf0bcc8f543194b85b39cdef84d74c838d932aacfae302ebd not found: ID does not exist" Nov 04 01:33:37.666573 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.666538 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j"] Nov 04 01:33:37.672017 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:37.671989 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98f846-776865ddf5-8ns7j"] Nov 04 01:33:38.506242 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:38.506207 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f1f1d7-b68a-489c-8914-6525a4079c6a" path="/var/lib/kubelet/pods/08f1f1d7-b68a-489c-8914-6525a4079c6a/volumes" Nov 04 01:33:39.276631 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:39.276598 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7"] Nov 04 01:33:39.276861 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:39.276834 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" podUID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" containerName="istio-proxy" containerID="cri-o://62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590" gracePeriod=30 Nov 04 01:33:39.763698 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:39.763662 5786 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:39.764051 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:39.763716 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" podUID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:42.061850 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:42.061789 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z"] Nov 04 01:33:42.162284 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:42.162245 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8"] Nov 04 01:33:44.533728 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.533703 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:33:44.579539 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.579442 5786 generic.go:356] "Generic (PLEG): container finished" podID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" containerID="62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590" exitCode=0 Nov 04 01:33:44.579539 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.579496 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" Nov 04 01:33:44.579747 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.579538 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" event={"ID":"3a38ad2e-6b50-457c-874b-b4b7944c63c2","Type":"ContainerDied","Data":"62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590"} Nov 04 01:33:44.579747 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.579606 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7" event={"ID":"3a38ad2e-6b50-457c-874b-b4b7944c63c2","Type":"ContainerDied","Data":"08ec3b7b41d1806c7c6a95c667ecb033ed91779254e5ca670a306044ede0993d"} Nov 04 01:33:44.579747 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.579640 5786 scope.go:122] "RemoveContainer" containerID="62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590" Nov 04 01:33:44.588808 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.588785 5786 scope.go:122] "RemoveContainer" containerID="62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590" Nov 04 01:33:44.589095 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:44.589075 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590\": container with ID starting with 62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590 not found: ID does not exist" containerID="62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590" Nov 04 01:33:44.589190 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.589105 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590"} err="failed to get container status \"62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590\": rpc error: code = NotFound desc = could not find container \"62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590\": container with ID starting with 62a9d99a5c8d4b95b8d9470320db83da206f46b8ac0ed9b33d16a506da4e6590 not found: ID does not exist" Nov 04 01:33:44.660844 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.660818 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-podinfo\") pod \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " Nov 04 01:33:44.660959 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.660855 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-token\" (UniqueName: \"kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-token\") pod \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " Nov 04 01:33:44.660959 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.660902 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-credential-socket\") pod \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " Nov 04 01:33:44.661054 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.660956 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-kube-api-access-vjwkl\" (UniqueName: \"kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-kube-api-access-vjwkl\") pod \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " Nov 04 01:33:44.661054 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661001 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-socket\") pod \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " Nov 04 01:33:44.661054 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661044 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-data\") pod \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " Nov 04 01:33:44.661234 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661156 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-envoy\") pod \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " Nov 04 01:33:44.661290 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661231 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istiod-ca-cert\") pod \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " Nov 04 01:33:44.661290 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661262 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-certs\") pod \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\" (UID: \"3a38ad2e-6b50-457c-874b-b4b7944c63c2\") " Nov 04 01:33:44.661404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661267 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-credential-socket" pod "3a38ad2e-6b50-457c-874b-b4b7944c63c2" (UID: "3a38ad2e-6b50-457c-874b-b4b7944c63c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:44.661404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661335 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-data" pod "3a38ad2e-6b50-457c-874b-b4b7944c63c2" (UID: "3a38ad2e-6b50-457c-874b-b4b7944c63c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:44.661404 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661350 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-socket" pod "3a38ad2e-6b50-457c-874b-b4b7944c63c2" (UID: "3a38ad2e-6b50-457c-874b-b4b7944c63c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:44.661665 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661641 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-certs" pod "3a38ad2e-6b50-457c-874b-b4b7944c63c2" (UID: "3a38ad2e-6b50-457c-874b-b4b7944c63c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:44.661743 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661646 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istiod-ca-cert" pod "3a38ad2e-6b50-457c-874b-b4b7944c63c2" (UID: "3a38ad2e-6b50-457c-874b-b4b7944c63c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:44.661743 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661681 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.661743 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661704 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.661743 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.661722 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.663300 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.663272 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-podinfo" pod "3a38ad2e-6b50-457c-874b-b4b7944c63c2" (UID: "3a38ad2e-6b50-457c-874b-b4b7944c63c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:44.663389 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.663318 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-envoy" pod "3a38ad2e-6b50-457c-874b-b4b7944c63c2" (UID: "3a38ad2e-6b50-457c-874b-b4b7944c63c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:44.663389 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.663337 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-token" pod "3a38ad2e-6b50-457c-874b-b4b7944c63c2" (UID: "3a38ad2e-6b50-457c-874b-b4b7944c63c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:44.663548 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.663522 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-kube-api-access-vjwkl" pod "3a38ad2e-6b50-457c-874b-b4b7944c63c2" (UID: "3a38ad2e-6b50-457c-874b-b4b7944c63c2"). InnerVolumeSpecName "kube-api-access-vjwkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:44.762782 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.762746 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.762782 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.762779 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.762984 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.762795 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjwkl\" (UniqueName: \"kubernetes.io/projected/3a38ad2e-6b50-457c-874b-b4b7944c63c2-kube-api-access-vjwkl\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.762984 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.762810 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.762984 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.762819 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a38ad2e-6b50-457c-874b-b4b7944c63c2-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.762984 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.762828 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a38ad2e-6b50-457c-874b-b4b7944c63c2-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.996829 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:44.996796 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7"] Nov 04 01:33:45.000792 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:45.000760 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-hs2n7"] Nov 04 01:33:45.086416 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:45.086372 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:33:46.505365 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:46.505333 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" path="/var/lib/kubelet/pods/3a38ad2e-6b50-457c-874b-b4b7944c63c2/volumes" Nov 04 01:33:47.062861 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.062802 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" podUID="9840601d-5ec0-4a50-8088-64b0ac4fdc4e" containerName="payload-processing" containerID="cri-o://13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e" gracePeriod=25 Nov 04 01:33:47.163427 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.163379 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" podUID="028d9ee5-74e8-448d-b697-d636933c0643" containerName="payload-pre-processing" containerID="cri-o://0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b" gracePeriod=25 Nov 04 01:33:47.320782 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.320719 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" Nov 04 01:33:47.391581 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.391551 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-kube-api-access-4xm9k\" (UniqueName: \"kubernetes.io/projected/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-kube-api-access-4xm9k\") pod \"9840601d-5ec0-4a50-8088-64b0ac4fdc4e\" (UID: \"9840601d-5ec0-4a50-8088-64b0ac4fdc4e\") " Nov 04 01:33:47.391746 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.391681 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-plugins-config-volume\") pod \"9840601d-5ec0-4a50-8088-64b0ac4fdc4e\" (UID: \"9840601d-5ec0-4a50-8088-64b0ac4fdc4e\") " Nov 04 01:33:47.392025 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.392003 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-plugins-config-volume" pod "9840601d-5ec0-4a50-8088-64b0ac4fdc4e" (UID: "9840601d-5ec0-4a50-8088-64b0ac4fdc4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:47.393701 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.393673 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-kube-api-access-4xm9k" pod "9840601d-5ec0-4a50-8088-64b0ac4fdc4e" (UID: "9840601d-5ec0-4a50-8088-64b0ac4fdc4e"). InnerVolumeSpecName "kube-api-access-4xm9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:47.493465 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.493431 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xm9k\" (UniqueName: \"kubernetes.io/projected/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-kube-api-access-4xm9k\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:47.493465 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.493459 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9840601d-5ec0-4a50-8088-64b0ac4fdc4e-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:47.596688 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.596652 5786 generic.go:356] "Generic (PLEG): container finished" podID="9840601d-5ec0-4a50-8088-64b0ac4fdc4e" containerID="13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e" exitCode=0 Nov 04 01:33:47.597174 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.596700 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" Nov 04 01:33:47.597174 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.596734 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" event={"ID":"9840601d-5ec0-4a50-8088-64b0ac4fdc4e","Type":"ContainerDied","Data":"13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e"} Nov 04 01:33:47.597174 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.596798 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z" event={"ID":"9840601d-5ec0-4a50-8088-64b0ac4fdc4e","Type":"ContainerDied","Data":"59acb7c1da1a2c40992e9dc88dfb2ee9e56b3c9191bf78365d7c69fb1fe119a3"} Nov 04 01:33:47.597174 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.596831 5786 scope.go:122] "RemoveContainer" containerID="13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e" Nov 04 01:33:47.607553 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.607531 5786 scope.go:122] "RemoveContainer" containerID="13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e" Nov 04 01:33:47.607846 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:33:47.607826 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e\": container with ID starting with 13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e not found: ID does not exist" containerID="13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e" Nov 04 01:33:47.607913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.607856 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e"} err="failed to get container status \"13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e\": rpc error: code = NotFound desc = could not find container \"13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e\": container with ID starting with 13f71ce11a322faca7331d3a97752453d8fe82c5b12b05a043e664e8c9d1af6e not found: ID does not exist" Nov 04 01:33:47.755607 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.755576 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z"] Nov 04 01:33:47.758782 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.758752 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e210ef-566b86cd74-cc66z"] Nov 04 01:33:47.874255 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:47.874140 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:33:48.504498 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:48.504463 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9840601d-5ec0-4a50-8088-64b0ac4fdc4e" path="/var/lib/kubelet/pods/9840601d-5ec0-4a50-8088-64b0ac4fdc4e/volumes" Nov 04 01:33:48.761426 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:48.761333 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" podUID="028d9ee5-74e8-448d-b697-d636933c0643" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:33:50.461735 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.461679 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" containerID="cri-o://c22cb6aa9cd2a7cbb063d334727812a0f14dac19c7fc5f8969582abbfabc60b2" gracePeriod=45 Nov 04 01:33:50.613266 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.613232 5786 generic.go:356] "Generic (PLEG): container finished" podID="c7102928-5338-4f63-950b-0900b80b07d3" containerID="c22cb6aa9cd2a7cbb063d334727812a0f14dac19c7fc5f8969582abbfabc60b2" exitCode=0 Nov 04 01:33:50.613427 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.613307 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" event={"ID":"c7102928-5338-4f63-950b-0900b80b07d3","Type":"ContainerDied","Data":"c22cb6aa9cd2a7cbb063d334727812a0f14dac19c7fc5f8969582abbfabc60b2"} Nov 04 01:33:50.716559 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.716507 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:33:50.828682 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.828644 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7102928-5338-4f63-950b-0900b80b07d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7102928-5338-4f63-950b-0900b80b07d3-tls-certs\") pod \"c7102928-5338-4f63-950b-0900b80b07d3\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " Nov 04 01:33:50.828851 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.828716 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-tmp-dir\") pod \"c7102928-5338-4f63-950b-0900b80b07d3\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " Nov 04 01:33:50.828851 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.828770 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-home\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-home\") pod \"c7102928-5338-4f63-950b-0900b80b07d3\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " Nov 04 01:33:50.828986 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.828927 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-model-cache\") pod \"c7102928-5338-4f63-950b-0900b80b07d3\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " Nov 04 01:33:50.828986 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.828964 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-dshm\") pod \"c7102928-5338-4f63-950b-0900b80b07d3\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " Nov 04 01:33:50.829081 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.828993 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7102928-5338-4f63-950b-0900b80b07d3-kube-api-access-qztnq\" (UniqueName: \"kubernetes.io/projected/c7102928-5338-4f63-950b-0900b80b07d3-kube-api-access-qztnq\") pod \"c7102928-5338-4f63-950b-0900b80b07d3\" (UID: \"c7102928-5338-4f63-950b-0900b80b07d3\") " Nov 04 01:33:50.829081 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.829007 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-tmp-dir" pod "c7102928-5338-4f63-950b-0900b80b07d3" (UID: "c7102928-5338-4f63-950b-0900b80b07d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:50.829405 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.829384 5786 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-tmp-dir\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.830090 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.830018 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-model-cache" pod "c7102928-5338-4f63-950b-0900b80b07d3" (UID: "c7102928-5338-4f63-950b-0900b80b07d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:50.831085 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.830552 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-home" pod "c7102928-5338-4f63-950b-0900b80b07d3" (UID: "c7102928-5338-4f63-950b-0900b80b07d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:50.833332 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.833305 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7102928-5338-4f63-950b-0900b80b07d3-tls-certs" pod "c7102928-5338-4f63-950b-0900b80b07d3" (UID: "c7102928-5338-4f63-950b-0900b80b07d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:50.833722 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.833687 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7102928-5338-4f63-950b-0900b80b07d3-kube-api-access-qztnq" pod "c7102928-5338-4f63-950b-0900b80b07d3" (UID: "c7102928-5338-4f63-950b-0900b80b07d3"). InnerVolumeSpecName "kube-api-access-qztnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:50.834016 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.833992 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-dshm" pod "c7102928-5338-4f63-950b-0900b80b07d3" (UID: "c7102928-5338-4f63-950b-0900b80b07d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:50.930203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.930171 5786 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-home\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.930203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.930197 5786 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-model-cache\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.930203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.930208 5786 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7102928-5338-4f63-950b-0900b80b07d3-dshm\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.930505 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.930223 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qztnq\" (UniqueName: \"kubernetes.io/projected/c7102928-5338-4f63-950b-0900b80b07d3-kube-api-access-qztnq\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.930505 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:50.930234 5786 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7102928-5338-4f63-950b-0900b80b07d3-tls-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.619246 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:51.619214 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" Nov 04 01:33:51.619736 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:51.619256 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck" event={"ID":"c7102928-5338-4f63-950b-0900b80b07d3","Type":"ContainerDied","Data":"be725acd558a8456901783734c04d80b045c34e2c79b2d99414c4567b5af05bd"} Nov 04 01:33:51.619736 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:51.619298 5786 scope.go:122] "RemoveContainer" containerID="c22cb6aa9cd2a7cbb063d334727812a0f14dac19c7fc5f8969582abbfabc60b2" Nov 04 01:33:51.777695 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:51.777665 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck"] Nov 04 01:33:51.824645 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:51.824611 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/premium-w5-e210ef-kserve-66cccd884b-9m9ck"] Nov 04 01:33:52.504542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:52.504506 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7102928-5338-4f63-950b-0900b80b07d3" path="/var/lib/kubelet/pods/c7102928-5338-4f63-950b-0900b80b07d3/volumes" Nov 04 01:33:58.762136 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:33:58.762086 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" podUID="028d9ee5-74e8-448d-b697-d636933c0643" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:34:02.248111 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:02.248069 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n"] Nov 04 01:34:02.248622 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:02.248286 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerName="istio-proxy" containerID="cri-o://ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb" gracePeriod=30 Nov 04 01:34:02.296674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:02.296640 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw"] Nov 04 01:34:02.296915 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:02.296891 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerName="istio-proxy" containerID="cri-o://fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e" gracePeriod=30 Nov 04 01:34:02.297948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:02.297920 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7"] Nov 04 01:34:02.298137 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:02.298100 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" podUID="4faf119c-25e3-4f4e-80d7-998f0f46a492" containerName="istio-proxy" containerID="cri-o://d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f" gracePeriod=30 Nov 04 01:34:02.344709 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:02.344677 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd"] Nov 04 01:34:02.344904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:02.344882 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" containerName="istio-proxy" containerID="cri-o://c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b" gracePeriod=30 Nov 04 01:34:04.999562 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:04.999524 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qmg2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:05.000055 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:04.999602 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:05.101884 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.101851 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq"] Nov 04 01:34:05.102155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.102100 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" podUID="b8bf9c7a-9890-4a39-98eb-1a2044daf167" containerName="maas-api" containerID="cri-o://2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea" gracePeriod=30 Nov 04 01:34:05.376302 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.376270 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:34:05.477477 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.477448 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-maas-api-tls\") pod \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " Nov 04 01:34:05.477661 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.477504 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8bf9c7a-9890-4a39-98eb-1a2044daf167-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8bf9c7a-9890-4a39-98eb-1a2044daf167-openshift-service-ca\") pod \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " Nov 04 01:34:05.477661 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.477611 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-metrics-certs\") pod \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " Nov 04 01:34:05.477661 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.477635 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8bf9c7a-9890-4a39-98eb-1a2044daf167-kube-api-access-lbv5q\" (UniqueName: \"kubernetes.io/projected/b8bf9c7a-9890-4a39-98eb-1a2044daf167-kube-api-access-lbv5q\") pod \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\" (UID: \"b8bf9c7a-9890-4a39-98eb-1a2044daf167\") " Nov 04 01:34:05.477917 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.477893 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8bf9c7a-9890-4a39-98eb-1a2044daf167-openshift-service-ca" pod "b8bf9c7a-9890-4a39-98eb-1a2044daf167" (UID: "b8bf9c7a-9890-4a39-98eb-1a2044daf167"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:05.479709 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.479683 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-metrics-certs" pod "b8bf9c7a-9890-4a39-98eb-1a2044daf167" (UID: "b8bf9c7a-9890-4a39-98eb-1a2044daf167"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:05.479709 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.479702 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-maas-api-tls" pod "b8bf9c7a-9890-4a39-98eb-1a2044daf167" (UID: "b8bf9c7a-9890-4a39-98eb-1a2044daf167"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:05.479864 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.479814 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8bf9c7a-9890-4a39-98eb-1a2044daf167-kube-api-access-lbv5q" pod "b8bf9c7a-9890-4a39-98eb-1a2044daf167" (UID: "b8bf9c7a-9890-4a39-98eb-1a2044daf167"). InnerVolumeSpecName "kube-api-access-lbv5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:05.579364 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.579281 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:05.579364 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.579310 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8bf9c7a-9890-4a39-98eb-1a2044daf167-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:05.579364 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.579321 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8bf9c7a-9890-4a39-98eb-1a2044daf167-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:05.579364 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.579329 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbv5q\" (UniqueName: \"kubernetes.io/projected/b8bf9c7a-9890-4a39-98eb-1a2044daf167-kube-api-access-lbv5q\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:05.718246 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.718210 5786 generic.go:356] "Generic (PLEG): container finished" podID="b8bf9c7a-9890-4a39-98eb-1a2044daf167" containerID="2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea" exitCode=0 Nov 04 01:34:05.718452 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.718254 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" Nov 04 01:34:05.718452 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.718294 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" event={"ID":"b8bf9c7a-9890-4a39-98eb-1a2044daf167","Type":"ContainerDied","Data":"2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea"} Nov 04 01:34:05.718452 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.718357 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq" event={"ID":"b8bf9c7a-9890-4a39-98eb-1a2044daf167","Type":"ContainerDied","Data":"b92432604c00fa24c224ca3194a49188dd19cdb84c894a23ed073105aba3b876"} Nov 04 01:34:05.718452 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.718387 5786 scope.go:122] "RemoveContainer" containerID="2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea" Nov 04 01:34:05.727999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.727978 5786 scope.go:122] "RemoveContainer" containerID="2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea" Nov 04 01:34:05.728305 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:34:05.728286 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea\": container with ID starting with 2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea not found: ID does not exist" containerID="2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea" Nov 04 01:34:05.728382 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.728313 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea"} err="failed to get container status \"2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea\": rpc error: code = NotFound desc = could not find container \"2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea\": container with ID starting with 2b5484960ea91e9f4f05c3cde9888390b1914211025fd8b241b805b8bb2dc7ea not found: ID does not exist" Nov 04 01:34:05.876036 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.876000 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq"] Nov 04 01:34:05.923767 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:05.923732 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e210ef-5f8f6f9d6f-9rrzq"] Nov 04 01:34:06.504372 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:06.504341 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8bf9c7a-9890-4a39-98eb-1a2044daf167" path="/var/lib/kubelet/pods/b8bf9c7a-9890-4a39-98eb-1a2044daf167/volumes" Nov 04 01:34:07.338658 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.338615 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9d8xd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:07.338812 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.338690 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:07.502209 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.497608 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" Nov 04 01:34:07.555999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.555972 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:34:07.602142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.602071 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-certs\") pod \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " Nov 04 01:34:07.602336 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.602171 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istiod-ca-cert\") pod \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " Nov 04 01:34:07.602336 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.602251 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028d9ee5-74e8-448d-b697-d636933c0643-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/028d9ee5-74e8-448d-b697-d636933c0643-plugins-config-volume\") pod \"028d9ee5-74e8-448d-b697-d636933c0643\" (UID: \"028d9ee5-74e8-448d-b697-d636933c0643\") " Nov 04 01:34:07.602336 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.602290 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028d9ee5-74e8-448d-b697-d636933c0643-kube-api-access-gv5z5\" (UniqueName: \"kubernetes.io/projected/028d9ee5-74e8-448d-b697-d636933c0643-kube-api-access-gv5z5\") pod \"028d9ee5-74e8-448d-b697-d636933c0643\" (UID: \"028d9ee5-74e8-448d-b697-d636933c0643\") " Nov 04 01:34:07.602336 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.602321 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-envoy\") pod \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " Nov 04 01:34:07.602597 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.602361 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-token\" (UniqueName: \"kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-token\") pod \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " Nov 04 01:34:07.602597 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.602394 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-credential-socket\") pod \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " Nov 04 01:34:07.602597 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.602438 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-kube-api-access-m9lgq\" (UniqueName: \"kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-kube-api-access-m9lgq\") pod \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " Nov 04 01:34:07.602597 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.602472 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-podinfo\") pod \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " Nov 04 01:34:07.602597 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.602503 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-socket\") pod \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " Nov 04 01:34:07.602881 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.602599 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-data\") pod \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\" (UID: \"cad8d0d6-46c6-4df3-ae85-50ce778aa84c\") " Nov 04 01:34:07.603378 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.603274 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-data" pod "cad8d0d6-46c6-4df3-ae85-50ce778aa84c" (UID: "cad8d0d6-46c6-4df3-ae85-50ce778aa84c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.604864 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.603873 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028d9ee5-74e8-448d-b697-d636933c0643-plugins-config-volume" pod "028d9ee5-74e8-448d-b697-d636933c0643" (UID: "028d9ee5-74e8-448d-b697-d636933c0643"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:07.604864 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.604443 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-certs" pod "cad8d0d6-46c6-4df3-ae85-50ce778aa84c" (UID: "cad8d0d6-46c6-4df3-ae85-50ce778aa84c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.604864 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.604665 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-socket" pod "cad8d0d6-46c6-4df3-ae85-50ce778aa84c" (UID: "cad8d0d6-46c6-4df3-ae85-50ce778aa84c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.604864 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.604783 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istiod-ca-cert" pod "cad8d0d6-46c6-4df3-ae85-50ce778aa84c" (UID: "cad8d0d6-46c6-4df3-ae85-50ce778aa84c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:07.604864 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.604832 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-credential-socket" pod "cad8d0d6-46c6-4df3-ae85-50ce778aa84c" (UID: "cad8d0d6-46c6-4df3-ae85-50ce778aa84c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.607925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.607897 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028d9ee5-74e8-448d-b697-d636933c0643-kube-api-access-gv5z5" pod "028d9ee5-74e8-448d-b697-d636933c0643" (UID: "028d9ee5-74e8-448d-b697-d636933c0643"). InnerVolumeSpecName "kube-api-access-gv5z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:07.610228 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.610191 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-token" pod "cad8d0d6-46c6-4df3-ae85-50ce778aa84c" (UID: "cad8d0d6-46c6-4df3-ae85-50ce778aa84c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:07.610331 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.610269 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-kube-api-access-m9lgq" pod "cad8d0d6-46c6-4df3-ae85-50ce778aa84c" (UID: "cad8d0d6-46c6-4df3-ae85-50ce778aa84c"). InnerVolumeSpecName "kube-api-access-m9lgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:07.619651 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.616213 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-envoy" pod "cad8d0d6-46c6-4df3-ae85-50ce778aa84c" (UID: "cad8d0d6-46c6-4df3-ae85-50ce778aa84c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.619651 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.616216 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-podinfo" pod "cad8d0d6-46c6-4df3-ae85-50ce778aa84c" (UID: "cad8d0d6-46c6-4df3-ae85-50ce778aa84c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:07.640189 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.640159 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:34:07.642821 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.642787 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:34:07.647823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.647797 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:34:07.704170 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704066 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-podinfo\") pod \"fad79149-b0b3-4a22-a116-8bc3ca56f325\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " Nov 04 01:34:07.704170 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704105 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-envoy\") pod \"fad79149-b0b3-4a22-a116-8bc3ca56f325\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " Nov 04 01:34:07.704170 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704166 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2502a53-973b-4987-9890-f1172bf9b84d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2502a53-973b-4987-9890-f1172bf9b84d-istio-podinfo\") pod \"f2502a53-973b-4987-9890-f1172bf9b84d\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " Nov 04 01:34:07.704432 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704271 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-token\" (UniqueName: \"kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-token\") pod \"4faf119c-25e3-4f4e-80d7-998f0f46a492\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " Nov 04 01:34:07.704432 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704306 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2502a53-973b-4987-9890-f1172bf9b84d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2502a53-973b-4987-9890-f1172bf9b84d-istiod-ca-cert\") pod \"f2502a53-973b-4987-9890-f1172bf9b84d\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " Nov 04 01:34:07.704432 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704326 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-kube-api-access-qxddl\" (UniqueName: \"kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-kube-api-access-qxddl\") pod \"fad79149-b0b3-4a22-a116-8bc3ca56f325\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " Nov 04 01:34:07.704432 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704355 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-credential-socket\") pod \"f2502a53-973b-4987-9890-f1172bf9b84d\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " Nov 04 01:34:07.704432 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704418 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-istio-token\" (UniqueName: \"kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-istio-token\") pod \"f2502a53-973b-4987-9890-f1172bf9b84d\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " Nov 04 01:34:07.704710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704453 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-socket\") pod \"f2502a53-973b-4987-9890-f1172bf9b84d\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " Nov 04 01:34:07.704710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704482 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-kube-api-access-7rtn4\" (UniqueName: \"kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-kube-api-access-7rtn4\") pod \"f2502a53-973b-4987-9890-f1172bf9b84d\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " Nov 04 01:34:07.704710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704512 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-socket\") pod \"fad79149-b0b3-4a22-a116-8bc3ca56f325\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " Nov 04 01:34:07.704710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704541 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-data\") pod \"4faf119c-25e3-4f4e-80d7-998f0f46a492\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " Nov 04 01:34:07.704710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704582 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-podinfo\") pod \"4faf119c-25e3-4f4e-80d7-998f0f46a492\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " Nov 04 01:34:07.704710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704647 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-envoy\") pod \"f2502a53-973b-4987-9890-f1172bf9b84d\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " Nov 04 01:34:07.704710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704676 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-kube-api-access-vp4pb\" (UniqueName: \"kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-kube-api-access-vp4pb\") pod \"4faf119c-25e3-4f4e-80d7-998f0f46a492\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " Nov 04 01:34:07.705089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704719 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-envoy\") pod \"4faf119c-25e3-4f4e-80d7-998f0f46a492\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " Nov 04 01:34:07.705089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704751 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-socket" pod "f2502a53-973b-4987-9890-f1172bf9b84d" (UID: "f2502a53-973b-4987-9890-f1172bf9b84d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.705089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704766 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2502a53-973b-4987-9890-f1172bf9b84d-istiod-ca-cert" pod "f2502a53-973b-4987-9890-f1172bf9b84d" (UID: "f2502a53-973b-4987-9890-f1172bf9b84d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:07.705089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704762 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-certs\") pod \"f2502a53-973b-4987-9890-f1172bf9b84d\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " Nov 04 01:34:07.705089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704830 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad79149-b0b3-4a22-a116-8bc3ca56f325-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad79149-b0b3-4a22-a116-8bc3ca56f325-istiod-ca-cert\") pod \"fad79149-b0b3-4a22-a116-8bc3ca56f325\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " Nov 04 01:34:07.705089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704875 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-data\") pod \"f2502a53-973b-4987-9890-f1172bf9b84d\" (UID: \"f2502a53-973b-4987-9890-f1172bf9b84d\") " Nov 04 01:34:07.705089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704938 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-certs\") pod \"4faf119c-25e3-4f4e-80d7-998f0f46a492\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " Nov 04 01:34:07.705089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704977 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-credential-socket" pod "f2502a53-973b-4987-9890-f1172bf9b84d" (UID: "f2502a53-973b-4987-9890-f1172bf9b84d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.705089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.704981 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-socket\") pod \"4faf119c-25e3-4f4e-80d7-998f0f46a492\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " Nov 04 01:34:07.705089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705049 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-credential-socket\") pod \"4faf119c-25e3-4f4e-80d7-998f0f46a492\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " Nov 04 01:34:07.705632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705169 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-token\" (UniqueName: \"kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-token\") pod \"fad79149-b0b3-4a22-a116-8bc3ca56f325\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " Nov 04 01:34:07.705632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705220 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-certs\") pod \"fad79149-b0b3-4a22-a116-8bc3ca56f325\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " Nov 04 01:34:07.705632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705266 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad79149-b0b3-4a22-a116-8bc3ca56f325-istiod-ca-cert" pod "fad79149-b0b3-4a22-a116-8bc3ca56f325" (UID: "fad79149-b0b3-4a22-a116-8bc3ca56f325"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:07.705632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705277 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-data\") pod \"fad79149-b0b3-4a22-a116-8bc3ca56f325\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " Nov 04 01:34:07.705632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705302 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-socket" pod "4faf119c-25e3-4f4e-80d7-998f0f46a492" (UID: "4faf119c-25e3-4f4e-80d7-998f0f46a492"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.705632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705326 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-credential-socket\") pod \"fad79149-b0b3-4a22-a116-8bc3ca56f325\" (UID: \"fad79149-b0b3-4a22-a116-8bc3ca56f325\") " Nov 04 01:34:07.705632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705326 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-certs" pod "f2502a53-973b-4987-9890-f1172bf9b84d" (UID: "f2502a53-973b-4987-9890-f1172bf9b84d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.705632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705363 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4faf119c-25e3-4f4e-80d7-998f0f46a492-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4faf119c-25e3-4f4e-80d7-998f0f46a492-istiod-ca-cert\") pod \"4faf119c-25e3-4f4e-80d7-998f0f46a492\" (UID: \"4faf119c-25e3-4f4e-80d7-998f0f46a492\") " Nov 04 01:34:07.705632 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705550 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-data" pod "f2502a53-973b-4987-9890-f1172bf9b84d" (UID: "f2502a53-973b-4987-9890-f1172bf9b84d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.706089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705722 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-certs" pod "fad79149-b0b3-4a22-a116-8bc3ca56f325" (UID: "fad79149-b0b3-4a22-a116-8bc3ca56f325"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.706089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705891 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-socket" pod "fad79149-b0b3-4a22-a116-8bc3ca56f325" (UID: "fad79149-b0b3-4a22-a116-8bc3ca56f325"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.706089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705894 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/028d9ee5-74e8-448d-b697-d636933c0643-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705926 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2502a53-973b-4987-9890-f1172bf9b84d-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705934 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4faf119c-25e3-4f4e-80d7-998f0f46a492-istiod-ca-cert" pod "4faf119c-25e3-4f4e-80d7-998f0f46a492" (UID: "4faf119c-25e3-4f4e-80d7-998f0f46a492"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:07.706089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705941 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705979 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv5z5\" (UniqueName: \"kubernetes.io/projected/028d9ee5-74e8-448d-b697-d636933c0643-kube-api-access-gv5z5\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.705999 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706015 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706031 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706089 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706047 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9lgq\" (UniqueName: \"kubernetes.io/projected/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-kube-api-access-m9lgq\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706062 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706146 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706161 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706176 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706189 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706204 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad79149-b0b3-4a22-a116-8bc3ca56f325-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706221 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706235 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706249 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706263 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706277 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad8d0d6-46c6-4df3-ae85-50ce778aa84c-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706158 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-credential-socket" pod "4faf119c-25e3-4f4e-80d7-998f0f46a492" (UID: "4faf119c-25e3-4f4e-80d7-998f0f46a492"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706259 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-certs" pod "4faf119c-25e3-4f4e-80d7-998f0f46a492" (UID: "4faf119c-25e3-4f4e-80d7-998f0f46a492"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706381 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-data" pod "fad79149-b0b3-4a22-a116-8bc3ca56f325" (UID: "fad79149-b0b3-4a22-a116-8bc3ca56f325"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706478 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-credential-socket" pod "fad79149-b0b3-4a22-a116-8bc3ca56f325" (UID: "fad79149-b0b3-4a22-a116-8bc3ca56f325"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.706712 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.706498 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-data" pod "4faf119c-25e3-4f4e-80d7-998f0f46a492" (UID: "4faf119c-25e3-4f4e-80d7-998f0f46a492"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.707703 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.707387 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-istio-token" pod "f2502a53-973b-4987-9890-f1172bf9b84d" (UID: "f2502a53-973b-4987-9890-f1172bf9b84d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:07.707703 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.707444 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-token" pod "4faf119c-25e3-4f4e-80d7-998f0f46a492" (UID: "4faf119c-25e3-4f4e-80d7-998f0f46a492"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:07.708395 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.708374 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-podinfo" pod "fad79149-b0b3-4a22-a116-8bc3ca56f325" (UID: "fad79149-b0b3-4a22-a116-8bc3ca56f325"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:07.708703 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.708668 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2502a53-973b-4987-9890-f1172bf9b84d-istio-podinfo" pod "f2502a53-973b-4987-9890-f1172bf9b84d" (UID: "f2502a53-973b-4987-9890-f1172bf9b84d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:07.708804 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.708763 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-kube-api-access-vp4pb" pod "4faf119c-25e3-4f4e-80d7-998f0f46a492" (UID: "4faf119c-25e3-4f4e-80d7-998f0f46a492"). InnerVolumeSpecName "kube-api-access-vp4pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:07.708884 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.708802 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-envoy" pod "fad79149-b0b3-4a22-a116-8bc3ca56f325" (UID: "fad79149-b0b3-4a22-a116-8bc3ca56f325"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.708950 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.708905 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-kube-api-access-qxddl" pod "fad79149-b0b3-4a22-a116-8bc3ca56f325" (UID: "fad79149-b0b3-4a22-a116-8bc3ca56f325"). InnerVolumeSpecName "kube-api-access-qxddl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:07.709260 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.709233 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-podinfo" pod "4faf119c-25e3-4f4e-80d7-998f0f46a492" (UID: "4faf119c-25e3-4f4e-80d7-998f0f46a492"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:07.709392 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.709376 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-token" pod "fad79149-b0b3-4a22-a116-8bc3ca56f325" (UID: "fad79149-b0b3-4a22-a116-8bc3ca56f325"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:07.709702 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.709682 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-kube-api-access-7rtn4" pod "f2502a53-973b-4987-9890-f1172bf9b84d" (UID: "f2502a53-973b-4987-9890-f1172bf9b84d"). InnerVolumeSpecName "kube-api-access-7rtn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:07.709809 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.709755 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-envoy" pod "4faf119c-25e3-4f4e-80d7-998f0f46a492" (UID: "4faf119c-25e3-4f4e-80d7-998f0f46a492"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.709945 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.709932 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-envoy" pod "f2502a53-973b-4987-9890-f1172bf9b84d" (UID: "f2502a53-973b-4987-9890-f1172bf9b84d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:07.733678 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.733646 5786 generic.go:356] "Generic (PLEG): container finished" podID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerID="fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e" exitCode=0 Nov 04 01:34:07.733861 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.733693 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" Nov 04 01:34:07.733861 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.733730 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" event={"ID":"fad79149-b0b3-4a22-a116-8bc3ca56f325","Type":"ContainerDied","Data":"fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e"} Nov 04 01:34:07.733861 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.733790 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw" event={"ID":"fad79149-b0b3-4a22-a116-8bc3ca56f325","Type":"ContainerDied","Data":"3323d0654acbdf9cbaa5e45db915ac2f2c78dfabb4815031957008c074327052"} Nov 04 01:34:07.733861 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.733819 5786 scope.go:122] "RemoveContainer" containerID="fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e" Nov 04 01:34:07.735063 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.735041 5786 generic.go:356] "Generic (PLEG): container finished" podID="4faf119c-25e3-4f4e-80d7-998f0f46a492" containerID="d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f" exitCode=0 Nov 04 01:34:07.735168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.735082 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" Nov 04 01:34:07.735168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.735144 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" event={"ID":"4faf119c-25e3-4f4e-80d7-998f0f46a492","Type":"ContainerDied","Data":"d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f"} Nov 04 01:34:07.735299 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.735197 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7" event={"ID":"4faf119c-25e3-4f4e-80d7-998f0f46a492","Type":"ContainerDied","Data":"b27c4f97772a737ccd4703301745b363ba2c02b33a1b2384ff5096e53d5a6fd0"} Nov 04 01:34:07.736715 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.736683 5786 generic.go:356] "Generic (PLEG): container finished" podID="f2502a53-973b-4987-9890-f1172bf9b84d" containerID="c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b" exitCode=0 Nov 04 01:34:07.736838 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.736727 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" Nov 04 01:34:07.736838 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.736776 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" event={"ID":"f2502a53-973b-4987-9890-f1172bf9b84d","Type":"ContainerDied","Data":"c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b"} Nov 04 01:34:07.736838 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.736817 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd" event={"ID":"f2502a53-973b-4987-9890-f1172bf9b84d","Type":"ContainerDied","Data":"730d93406915ed51299a7260cd6d54f3e86a96c6661e1eab8c40d247810ba810"} Nov 04 01:34:07.738311 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.738289 5786 generic.go:356] "Generic (PLEG): container finished" podID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerID="ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb" exitCode=0 Nov 04 01:34:07.738425 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.738329 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" Nov 04 01:34:07.738425 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.738383 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" event={"ID":"cad8d0d6-46c6-4df3-ae85-50ce778aa84c","Type":"ContainerDied","Data":"ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb"} Nov 04 01:34:07.738559 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.738425 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n" event={"ID":"cad8d0d6-46c6-4df3-ae85-50ce778aa84c","Type":"ContainerDied","Data":"0b9be40cf4ad5d454f9431944d296fabb8cf3e1bfc62f59c0e2a39ceff093678"} Nov 04 01:34:07.746492 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.745842 5786 scope.go:122] "RemoveContainer" containerID="fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e" Nov 04 01:34:07.746492 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.745997 5786 generic.go:356] "Generic (PLEG): container finished" podID="028d9ee5-74e8-448d-b697-d636933c0643" containerID="0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b" exitCode=0 Nov 04 01:34:07.746492 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.746067 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" event={"ID":"028d9ee5-74e8-448d-b697-d636933c0643","Type":"ContainerDied","Data":"0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b"} Nov 04 01:34:07.746492 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.746081 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" Nov 04 01:34:07.746492 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.746108 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8" event={"ID":"028d9ee5-74e8-448d-b697-d636933c0643","Type":"ContainerDied","Data":"8041366582beaeb823852f2c4660f73153d494e34948f2bb19073f306cfb3e65"} Nov 04 01:34:07.746492 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:34:07.746438 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e\": container with ID starting with fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e not found: ID does not exist" containerID="fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e" Nov 04 01:34:07.746492 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.746476 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e"} err="failed to get container status \"fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e\": rpc error: code = NotFound desc = could not find container \"fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e\": container with ID starting with fb4daca9a95553053997821817d11b92df7006ebc4c8be72384d6c16521a606e not found: ID does not exist" Nov 04 01:34:07.746940 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.746500 5786 scope.go:122] "RemoveContainer" containerID="d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f" Nov 04 01:34:07.762806 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.762776 5786 scope.go:122] "RemoveContainer" containerID="d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f" Nov 04 01:34:07.764846 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:34:07.763214 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f\": container with ID starting with d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f not found: ID does not exist" containerID="d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f" Nov 04 01:34:07.764846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.763261 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f"} err="failed to get container status \"d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f\": rpc error: code = NotFound desc = could not find container \"d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f\": container with ID starting with d739a7a6e740266c6170c4043cbf7f4391c1dcb3209c77be33143177eae8cd4f not found: ID does not exist" Nov 04 01:34:07.764846 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.763287 5786 scope.go:122] "RemoveContainer" containerID="c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b" Nov 04 01:34:07.779028 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.779003 5786 scope.go:122] "RemoveContainer" containerID="c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b" Nov 04 01:34:07.779364 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:34:07.779342 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b\": container with ID starting with c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b not found: ID does not exist" containerID="c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b" Nov 04 01:34:07.779436 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.779374 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b"} err="failed to get container status \"c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b\": rpc error: code = NotFound desc = could not find container \"c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b\": container with ID starting with c1506ac4b0884001e6b44534404801688c55109cb65688289a354ac48b27601b not found: ID does not exist" Nov 04 01:34:07.779436 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.779391 5786 scope.go:122] "RemoveContainer" containerID="ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb" Nov 04 01:34:07.788260 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.788111 5786 scope.go:122] "RemoveContainer" containerID="ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb" Nov 04 01:34:07.788556 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:34:07.788525 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb\": container with ID starting with ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb not found: ID does not exist" containerID="ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb" Nov 04 01:34:07.788643 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.788568 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb"} err="failed to get container status \"ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb\": rpc error: code = NotFound desc = could not find container \"ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb\": container with ID starting with ee7fc2468717262e7621c1bdd1513b132efc9a19e431c5f0beef748f4f99bbdb not found: ID does not exist" Nov 04 01:34:07.788643 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.788595 5786 scope.go:122] "RemoveContainer" containerID="0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b" Nov 04 01:34:07.798234 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.798213 5786 scope.go:122] "RemoveContainer" containerID="0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b" Nov 04 01:34:07.798608 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:34:07.798581 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b\": container with ID starting with 0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b not found: ID does not exist" containerID="0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b" Nov 04 01:34:07.798707 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.798617 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b"} err="failed to get container status \"0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b\": rpc error: code = NotFound desc = could not find container \"0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b\": container with ID starting with 0509c5c53ee6d4f10bdf7434082b04b89ba8e09d9da283c91ef2eba876f4ca7b not found: ID does not exist" Nov 04 01:34:07.807231 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807205 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2502a53-973b-4987-9890-f1172bf9b84d-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807234 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807248 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxddl\" (UniqueName: \"kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-kube-api-access-qxddl\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807263 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807276 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtn4\" (UniqueName: \"kubernetes.io/projected/f2502a53-973b-4987-9890-f1172bf9b84d-kube-api-access-7rtn4\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807289 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807301 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807314 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807326 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2502a53-973b-4987-9890-f1172bf9b84d-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807339 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp4pb\" (UniqueName: \"kubernetes.io/projected/4faf119c-25e3-4f4e-80d7-998f0f46a492-kube-api-access-vp4pb\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807353 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807711 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807366 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807711 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807380 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faf119c-25e3-4f4e-80d7-998f0f46a492-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807711 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807394 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807711 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807407 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807711 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807420 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807711 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807434 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4faf119c-25e3-4f4e-80d7-998f0f46a492-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807711 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807448 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:07.807711 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:07.807461 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad79149-b0b3-4a22-a116-8bc3ca56f325-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:08.166243 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:08.166206 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7"] Nov 04 01:34:08.168244 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:08.168215 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pgvz7"] Nov 04 01:34:08.411098 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:08.411064 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw"] Nov 04 01:34:08.425760 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:08.425685 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mrdrw"] Nov 04 01:34:08.503897 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:08.503863 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4faf119c-25e3-4f4e-80d7-998f0f46a492" path="/var/lib/kubelet/pods/4faf119c-25e3-4f4e-80d7-998f0f46a492/volumes" Nov 04 01:34:08.504337 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:08.504323 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" path="/var/lib/kubelet/pods/fad79149-b0b3-4a22-a116-8bc3ca56f325/volumes" Nov 04 01:34:08.538057 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:08.538023 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd"] Nov 04 01:34:08.548945 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:08.548910 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9d8xd"] Nov 04 01:34:08.662762 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:08.662729 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n"] Nov 04 01:34:08.674568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:08.674539 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qmg2n"] Nov 04 01:34:08.780074 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:08.779998 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8"] Nov 04 01:34:08.789406 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:08.789382 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e210ef-577594f6b7-5hmn8"] Nov 04 01:34:10.303917 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:10.303882 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:34:10.504208 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:10.504175 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="028d9ee5-74e8-448d-b697-d636933c0643" path="/var/lib/kubelet/pods/028d9ee5-74e8-448d-b697-d636933c0643/volumes" Nov 04 01:34:10.504625 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:10.504613 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" path="/var/lib/kubelet/pods/cad8d0d6-46c6-4df3-ae85-50ce778aa84c/volumes" Nov 04 01:34:10.504999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:10.504987 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" path="/var/lib/kubelet/pods/f2502a53-973b-4987-9890-f1172bf9b84d/volumes" Nov 04 01:34:15.086797 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:34:15.086760 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:34:26.443081 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:26.443045 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:34:27.627645 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:27.627614 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj"] Nov 04 01:34:42.647646 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.647607 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" containerID="cri-o://d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11" gracePeriod=45 Nov 04 01:34:42.899950 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.899892 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:34:42.919483 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.919448 5786 generic.go:356] "Generic (PLEG): container finished" podID="e54f93ab-904a-4feb-a157-fa470223d756" containerID="d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11" exitCode=0 Nov 04 01:34:42.919636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.919485 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" Nov 04 01:34:42.919636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.919486 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" event={"ID":"e54f93ab-904a-4feb-a157-fa470223d756","Type":"ContainerDied","Data":"d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11"} Nov 04 01:34:42.919636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.919538 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" event={"ID":"e54f93ab-904a-4feb-a157-fa470223d756","Type":"ContainerDied","Data":"6c0704066a3036d080025a161227e2f59b4387422b28c5b6105260b1e17548d1"} Nov 04 01:34:42.919636 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.919563 5786 scope.go:122] "RemoveContainer" containerID="d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11" Nov 04 01:34:42.928669 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.928651 5786 scope.go:122] "RemoveContainer" containerID="d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11" Nov 04 01:34:42.928921 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:34:42.928904 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11\": container with ID starting with d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11 not found: ID does not exist" containerID="d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11" Nov 04 01:34:42.928967 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.928933 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11"} err="failed to get container status \"d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11\": rpc error: code = NotFound desc = could not find container \"d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11\": container with ID starting with d117ef22bb91b65144c0091b968b9bfa8045597fd0ba39fefb98ce05d3822d11 not found: ID does not exist" Nov 04 01:34:42.946500 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.946475 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-tmp-dir\") pod \"e54f93ab-904a-4feb-a157-fa470223d756\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " Nov 04 01:34:42.946601 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.946571 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-home\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-home\") pod \"e54f93ab-904a-4feb-a157-fa470223d756\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " Nov 04 01:34:42.946671 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.946600 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54f93ab-904a-4feb-a157-fa470223d756-kube-api-access-mczzd\" (UniqueName: \"kubernetes.io/projected/e54f93ab-904a-4feb-a157-fa470223d756-kube-api-access-mczzd\") pod \"e54f93ab-904a-4feb-a157-fa470223d756\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " Nov 04 01:34:42.946671 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.946648 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-model-cache\") pod \"e54f93ab-904a-4feb-a157-fa470223d756\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " Nov 04 01:34:42.946785 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.946671 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e54f93ab-904a-4feb-a157-fa470223d756-tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f93ab-904a-4feb-a157-fa470223d756-tls-certs\") pod \"e54f93ab-904a-4feb-a157-fa470223d756\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " Nov 04 01:34:42.946785 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.946697 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-dshm\") pod \"e54f93ab-904a-4feb-a157-fa470223d756\" (UID: \"e54f93ab-904a-4feb-a157-fa470223d756\") " Nov 04 01:34:42.946926 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.946761 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-tmp-dir" pod "e54f93ab-904a-4feb-a157-fa470223d756" (UID: "e54f93ab-904a-4feb-a157-fa470223d756"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:42.946926 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.946844 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-home" pod "e54f93ab-904a-4feb-a157-fa470223d756" (UID: "e54f93ab-904a-4feb-a157-fa470223d756"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:42.947043 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.946909 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-model-cache" pod "e54f93ab-904a-4feb-a157-fa470223d756" (UID: "e54f93ab-904a-4feb-a157-fa470223d756"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:42.947043 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.946996 5786 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-model-cache\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:42.947043 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.947014 5786 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-tmp-dir\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:42.947043 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.947034 5786 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-home\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:42.948874 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.948852 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54f93ab-904a-4feb-a157-fa470223d756-tls-certs" pod "e54f93ab-904a-4feb-a157-fa470223d756" (UID: "e54f93ab-904a-4feb-a157-fa470223d756"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:42.948957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.948895 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54f93ab-904a-4feb-a157-fa470223d756-kube-api-access-mczzd" pod "e54f93ab-904a-4feb-a157-fa470223d756" (UID: "e54f93ab-904a-4feb-a157-fa470223d756"). InnerVolumeSpecName "kube-api-access-mczzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:42.948957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:42.948895 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-dshm" pod "e54f93ab-904a-4feb-a157-fa470223d756" (UID: "e54f93ab-904a-4feb-a157-fa470223d756"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:43.048455 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:43.048415 5786 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f93ab-904a-4feb-a157-fa470223d756-tls-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.048455 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:43.048445 5786 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54f93ab-904a-4feb-a157-fa470223d756-dshm\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.048455 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:43.048459 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mczzd\" (UniqueName: \"kubernetes.io/projected/e54f93ab-904a-4feb-a157-fa470223d756-kube-api-access-mczzd\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.340435 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:43.340400 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj"] Nov 04 01:34:43.351630 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:43.351594 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj"] Nov 04 01:34:43.826567 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:43.826525 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/premium-w2-2fd27d-kserve-66df56bcf-b4vgj" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:34:44.511157 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:44.511096 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54f93ab-904a-4feb-a157-fa470223d756" path="/var/lib/kubelet/pods/e54f93ab-904a-4feb-a157-fa470223d756/volumes" Nov 04 01:34:45.086535 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:34:45.086502 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:34:53.153816 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:53.153769 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:34:54.529459 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.529419 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh"] Nov 04 01:34:54.529924 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.529684 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" podUID="a5958d8a-1f07-45d1-8410-57494d457efe" containerName="maas-api" containerID="cri-o://f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af" gracePeriod=30 Nov 04 01:34:54.821402 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.821370 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:34:54.862554 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.862521 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5958d8a-1f07-45d1-8410-57494d457efe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5958d8a-1f07-45d1-8410-57494d457efe-openshift-service-ca\") pod \"a5958d8a-1f07-45d1-8410-57494d457efe\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " Nov 04 01:34:54.862728 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.862588 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-metrics-certs\") pod \"a5958d8a-1f07-45d1-8410-57494d457efe\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " Nov 04 01:34:54.862772 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.862755 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5958d8a-1f07-45d1-8410-57494d457efe-kube-api-access-ljqwl\" (UniqueName: \"kubernetes.io/projected/a5958d8a-1f07-45d1-8410-57494d457efe-kube-api-access-ljqwl\") pod \"a5958d8a-1f07-45d1-8410-57494d457efe\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " Nov 04 01:34:54.862876 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.862863 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-maas-api-tls\") pod \"a5958d8a-1f07-45d1-8410-57494d457efe\" (UID: \"a5958d8a-1f07-45d1-8410-57494d457efe\") " Nov 04 01:34:54.862985 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.862963 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5958d8a-1f07-45d1-8410-57494d457efe-openshift-service-ca" pod "a5958d8a-1f07-45d1-8410-57494d457efe" (UID: "a5958d8a-1f07-45d1-8410-57494d457efe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:54.863167 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.863108 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5958d8a-1f07-45d1-8410-57494d457efe-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:54.864859 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.864828 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-maas-api-tls" pod "a5958d8a-1f07-45d1-8410-57494d457efe" (UID: "a5958d8a-1f07-45d1-8410-57494d457efe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:54.864979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.864903 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5958d8a-1f07-45d1-8410-57494d457efe-kube-api-access-ljqwl" pod "a5958d8a-1f07-45d1-8410-57494d457efe" (UID: "a5958d8a-1f07-45d1-8410-57494d457efe"). InnerVolumeSpecName "kube-api-access-ljqwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:54.864979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.864904 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-metrics-certs" pod "a5958d8a-1f07-45d1-8410-57494d457efe" (UID: "a5958d8a-1f07-45d1-8410-57494d457efe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:54.964543 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.964504 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:54.964543 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.964536 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5958d8a-1f07-45d1-8410-57494d457efe-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:54.964543 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.964546 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljqwl\" (UniqueName: \"kubernetes.io/projected/a5958d8a-1f07-45d1-8410-57494d457efe-kube-api-access-ljqwl\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:54.974982 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.974953 5786 generic.go:356] "Generic (PLEG): container finished" podID="a5958d8a-1f07-45d1-8410-57494d457efe" containerID="f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af" exitCode=0 Nov 04 01:34:54.975139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.975005 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" Nov 04 01:34:54.975139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.975014 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" event={"ID":"a5958d8a-1f07-45d1-8410-57494d457efe","Type":"ContainerDied","Data":"f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af"} Nov 04 01:34:54.975139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.975054 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh" event={"ID":"a5958d8a-1f07-45d1-8410-57494d457efe","Type":"ContainerDied","Data":"7d61296facaf915ba4a12fbfc58b58efdc8582bd018340f5f941f6915cfdf572"} Nov 04 01:34:54.975139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.975075 5786 scope.go:122] "RemoveContainer" containerID="f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af" Nov 04 01:34:54.984381 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.984364 5786 scope.go:122] "RemoveContainer" containerID="f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af" Nov 04 01:34:54.984652 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:34:54.984635 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af\": container with ID starting with f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af not found: ID does not exist" containerID="f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af" Nov 04 01:34:54.984718 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:54.984658 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af"} err="failed to get container status \"f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af\": rpc error: code = NotFound desc = could not find container \"f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af\": container with ID starting with f9869e13217433f5e2126daf05872a5632e866e7c4f259f2b1462a66a7d753af not found: ID does not exist" Nov 04 01:34:55.131824 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:55.131786 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh"] Nov 04 01:34:55.134048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:55.134022 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2fd27d-7f9d8594f4-v46vh"] Nov 04 01:34:56.503737 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:34:56.503710 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5958d8a-1f07-45d1-8410-57494d457efe" path="/var/lib/kubelet/pods/a5958d8a-1f07-45d1-8410-57494d457efe/volumes" Nov 04 01:35:04.210511 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:04.210468 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh"] Nov 04 01:35:04.210925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:04.210729 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerName="istio-proxy" containerID="cri-o://0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951" gracePeriod=30 Nov 04 01:35:04.218984 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:04.218953 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p"] Nov 04 01:35:04.219252 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:04.219208 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" podUID="705339b0-c4d3-46ad-b55e-919f4113cecc" containerName="istio-proxy" containerID="cri-o://1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72" gracePeriod=30 Nov 04 01:35:08.487081 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:08.487040 5786 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:35:08.487561 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:08.487098 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" podUID="705339b0-c4d3-46ad-b55e-919f4113cecc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:35:09.464328 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.464303 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:35:09.495992 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.495967 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:35:09.606185 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606148 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-data\") pod \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " Nov 04 01:35:09.606185 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606191 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-socket\") pod \"705339b0-c4d3-46ad-b55e-919f4113cecc\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " Nov 04 01:35:09.606452 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606214 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-kube-api-access-2gfhq\" (UniqueName: \"kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-kube-api-access-2gfhq\") pod \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " Nov 04 01:35:09.606452 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606243 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-data\") pod \"705339b0-c4d3-46ad-b55e-919f4113cecc\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " Nov 04 01:35:09.606452 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606269 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-credential-socket\") pod \"705339b0-c4d3-46ad-b55e-919f4113cecc\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " Nov 04 01:35:09.606452 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606293 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-socket\") pod \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " Nov 04 01:35:09.606452 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606438 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-data" pod "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" (UID: "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.606710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606521 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-certs\") pod \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " Nov 04 01:35:09.606710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606539 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-socket" pod "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" (UID: "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.606710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606550 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-podinfo\") pod \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " Nov 04 01:35:09.606710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606562 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-data" pod "705339b0-c4d3-46ad-b55e-919f4113cecc" (UID: "705339b0-c4d3-46ad-b55e-919f4113cecc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.606710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606547 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-socket" pod "705339b0-c4d3-46ad-b55e-919f4113cecc" (UID: "705339b0-c4d3-46ad-b55e-919f4113cecc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.606710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606592 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-kube-api-access-qhqb9\" (UniqueName: \"kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-kube-api-access-qhqb9\") pod \"705339b0-c4d3-46ad-b55e-919f4113cecc\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " Nov 04 01:35:09.606710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606610 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-credential-socket" pod "705339b0-c4d3-46ad-b55e-919f4113cecc" (UID: "705339b0-c4d3-46ad-b55e-919f4113cecc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.606710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606630 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-podinfo\") pod \"705339b0-c4d3-46ad-b55e-919f4113cecc\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " Nov 04 01:35:09.606710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606680 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705339b0-c4d3-46ad-b55e-919f4113cecc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/705339b0-c4d3-46ad-b55e-919f4113cecc-istiod-ca-cert\") pod \"705339b0-c4d3-46ad-b55e-919f4113cecc\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " Nov 04 01:35:09.606710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606713 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-envoy\") pod \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " Nov 04 01:35:09.607259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606736 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-certs" pod "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" (UID: "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.607259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606765 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-certs\") pod \"705339b0-c4d3-46ad-b55e-919f4113cecc\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " Nov 04 01:35:09.607259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606797 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-token\" (UniqueName: \"kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-token\") pod \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " Nov 04 01:35:09.607259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606841 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istiod-ca-cert\") pod \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " Nov 04 01:35:09.607259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606869 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-credential-socket\") pod \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\" (UID: \"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4\") " Nov 04 01:35:09.607259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606927 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-token\" (UniqueName: \"kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-token\") pod \"705339b0-c4d3-46ad-b55e-919f4113cecc\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " Nov 04 01:35:09.607259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.606956 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-envoy\") pod \"705339b0-c4d3-46ad-b55e-919f4113cecc\" (UID: \"705339b0-c4d3-46ad-b55e-919f4113cecc\") " Nov 04 01:35:09.607259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.607025 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705339b0-c4d3-46ad-b55e-919f4113cecc-istiod-ca-cert" pod "705339b0-c4d3-46ad-b55e-919f4113cecc" (UID: "705339b0-c4d3-46ad-b55e-919f4113cecc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:09.607674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.607287 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.607674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.607307 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.607674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.607321 5786 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-data\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.607674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.607335 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.607674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.607347 5786 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.607674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.607361 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.607674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.607375 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/705339b0-c4d3-46ad-b55e-919f4113cecc-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.607674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.607439 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-credential-socket" pod "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" (UID: "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.607674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.607530 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istiod-ca-cert" pod "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" (UID: "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:09.607674 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.607641 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-certs" pod "705339b0-c4d3-46ad-b55e-919f4113cecc" (UID: "705339b0-c4d3-46ad-b55e-919f4113cecc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.609223 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.609183 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-kube-api-access-2gfhq" pod "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" (UID: "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4"). InnerVolumeSpecName "kube-api-access-2gfhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:09.609223 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.609195 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-kube-api-access-qhqb9" pod "705339b0-c4d3-46ad-b55e-919f4113cecc" (UID: "705339b0-c4d3-46ad-b55e-919f4113cecc"). InnerVolumeSpecName "kube-api-access-qhqb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:09.609469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.609447 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-envoy" pod "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" (UID: "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.609554 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.609509 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-podinfo" pod "705339b0-c4d3-46ad-b55e-919f4113cecc" (UID: "705339b0-c4d3-46ad-b55e-919f4113cecc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:09.609744 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.609719 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-podinfo" pod "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" (UID: "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:09.609844 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.609813 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-envoy" pod "705339b0-c4d3-46ad-b55e-919f4113cecc" (UID: "705339b0-c4d3-46ad-b55e-919f4113cecc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.609903 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.609844 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-token" pod "705339b0-c4d3-46ad-b55e-919f4113cecc" (UID: "705339b0-c4d3-46ad-b55e-919f4113cecc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:09.610330 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.610307 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-token" pod "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" (UID: "eec1b4ed-2b7c-4c22-931a-d8b509afd5d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:09.708749 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.708711 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.708749 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.708741 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.708749 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.708751 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gfhq\" (UniqueName: \"kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-kube-api-access-2gfhq\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.708983 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.708761 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.708983 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.708770 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhqb9\" (UniqueName: \"kubernetes.io/projected/705339b0-c4d3-46ad-b55e-919f4113cecc-kube-api-access-qhqb9\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.708983 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.708779 5786 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/705339b0-c4d3-46ad-b55e-919f4113cecc-istio-podinfo\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.708983 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.708787 5786 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-envoy\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.708983 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.708796 5786 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/705339b0-c4d3-46ad-b55e-919f4113cecc-workload-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.708983 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.708803 5786 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istio-token\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.708983 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.708828 5786 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-istiod-ca-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.708983 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:09.708838 5786 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4-credential-socket\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:10.047028 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.046990 5786 generic.go:356] "Generic (PLEG): container finished" podID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerID="0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951" exitCode=0 Nov 04 01:35:10.047270 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.047040 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" Nov 04 01:35:10.047270 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.047075 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" event={"ID":"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4","Type":"ContainerDied","Data":"0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951"} Nov 04 01:35:10.047270 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.047158 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh" event={"ID":"eec1b4ed-2b7c-4c22-931a-d8b509afd5d4","Type":"ContainerDied","Data":"1f617400e88979b7c482fa9fe0c8f672acb1e494746bc64d06b884d249d4b18c"} Nov 04 01:35:10.047270 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.047188 5786 scope.go:122] "RemoveContainer" containerID="0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951" Nov 04 01:35:10.048375 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.048351 5786 generic.go:356] "Generic (PLEG): container finished" podID="705339b0-c4d3-46ad-b55e-919f4113cecc" containerID="1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72" exitCode=0 Nov 04 01:35:10.048440 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.048404 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" Nov 04 01:35:10.048440 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.048418 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" event={"ID":"705339b0-c4d3-46ad-b55e-919f4113cecc","Type":"ContainerDied","Data":"1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72"} Nov 04 01:35:10.048589 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.048458 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p" event={"ID":"705339b0-c4d3-46ad-b55e-919f4113cecc","Type":"ContainerDied","Data":"b1dee0fc99f8a7836b68c6da52927a6078ad607c97c70d67d00425d1195b66fc"} Nov 04 01:35:10.057570 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.057548 5786 scope.go:122] "RemoveContainer" containerID="0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951" Nov 04 01:35:10.057861 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:35:10.057837 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951\": container with ID starting with 0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951 not found: ID does not exist" containerID="0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951" Nov 04 01:35:10.057910 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.057874 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951"} err="failed to get container status \"0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951\": rpc error: code = NotFound desc = could not find container \"0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951\": container with ID starting with 0688411f4c6513131543a2853c21888b4efde66d6a6e1f744718e52dd16fc951 not found: ID does not exist" Nov 04 01:35:10.057910 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.057897 5786 scope.go:122] "RemoveContainer" containerID="1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72" Nov 04 01:35:10.066095 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.066076 5786 scope.go:122] "RemoveContainer" containerID="1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72" Nov 04 01:35:10.066382 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:35:10.066362 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72\": container with ID starting with 1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72 not found: ID does not exist" containerID="1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72" Nov 04 01:35:10.066458 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.066387 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72"} err="failed to get container status \"1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72\": rpc error: code = NotFound desc = could not find container \"1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72\": container with ID starting with 1faeb7b7c1b7e6338b5cac932faa476b49a6e0eec9b1ee9192231f44001aad72 not found: ID does not exist" Nov 04 01:35:10.207329 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.207297 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh"] Nov 04 01:35:10.255245 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.255212 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-t2mgh"] Nov 04 01:35:10.371974 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.368725 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p"] Nov 04 01:35:10.373768 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.373738 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-zlw8p"] Nov 04 01:35:10.504055 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.504020 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705339b0-c4d3-46ad-b55e-919f4113cecc" path="/var/lib/kubelet/pods/705339b0-c4d3-46ad-b55e-919f4113cecc/volumes" Nov 04 01:35:10.504483 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:10.504469 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" path="/var/lib/kubelet/pods/eec1b4ed-2b7c-4c22-931a-d8b509afd5d4/volumes" Nov 04 01:35:11.599017 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.598967 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-566cd577c-zqm98"] Nov 04 01:35:11.599550 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599507 5786 cpu_manager.go:414] "containerMap: removing container" podUID="ce0ed274-cc86-4ab4-b8fb-db31ee650267" containerName="maas-api" Nov 04 01:35:11.599550 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599527 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0ed274-cc86-4ab4-b8fb-db31ee650267" containerName="maas-api" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599559 5786 cpu_manager.go:414] "containerMap: removing container" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599567 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599577 5786 cpu_manager.go:414] "containerMap: removing container" podUID="9840601d-5ec0-4a50-8088-64b0ac4fdc4e" containerName="payload-processing" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599585 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9840601d-5ec0-4a50-8088-64b0ac4fdc4e" containerName="payload-processing" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599596 5786 cpu_manager.go:414] "containerMap: removing container" podUID="e496528c-9914-4f1f-bb29-f9d10ac5b3b2" containerName="payload-processing" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599604 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e496528c-9914-4f1f-bb29-f9d10ac5b3b2" containerName="payload-processing" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599614 5786 cpu_manager.go:414] "containerMap: removing container" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerName="istio-proxy" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599623 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerName="istio-proxy" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599639 5786 cpu_manager.go:414] "containerMap: removing container" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599646 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599656 5786 cpu_manager.go:414] "containerMap: removing container" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599663 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599676 5786 cpu_manager.go:414] "containerMap: removing container" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerName="istio-proxy" Nov 04 01:35:11.599683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599683 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599694 5786 cpu_manager.go:414] "containerMap: removing container" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599701 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599711 5786 cpu_manager.go:414] "containerMap: removing container" podUID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599718 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599729 5786 cpu_manager.go:414] "containerMap: removing container" podUID="a5958d8a-1f07-45d1-8410-57494d457efe" containerName="maas-api" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599736 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5958d8a-1f07-45d1-8410-57494d457efe" containerName="maas-api" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599745 5786 cpu_manager.go:414] "containerMap: removing container" podUID="fcd9752e-6a9b-470f-ad9d-938bca9a46cc" containerName="payload-pre-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599753 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd9752e-6a9b-470f-ad9d-938bca9a46cc" containerName="payload-pre-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599766 5786 cpu_manager.go:414] "containerMap: removing container" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599775 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599787 5786 cpu_manager.go:414] "containerMap: removing container" podUID="028d9ee5-74e8-448d-b697-d636933c0643" containerName="payload-pre-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599794 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028d9ee5-74e8-448d-b697-d636933c0643" containerName="payload-pre-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599806 5786 cpu_manager.go:414] "containerMap: removing container" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599813 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599823 5786 cpu_manager.go:414] "containerMap: removing container" podUID="8ece2c47-d09b-4355-8b9d-7df3a80de475" containerName="test-curl-7240-7b26ca" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599830 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ece2c47-d09b-4355-8b9d-7df3a80de475" containerName="test-curl-7240-7b26ca" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599844 5786 cpu_manager.go:414] "containerMap: removing container" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599851 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599861 5786 cpu_manager.go:414] "containerMap: removing container" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599868 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599877 5786 cpu_manager.go:414] "containerMap: removing container" podUID="4faf119c-25e3-4f4e-80d7-998f0f46a492" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599885 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4faf119c-25e3-4f4e-80d7-998f0f46a492" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599897 5786 cpu_manager.go:414] "containerMap: removing container" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599905 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599916 5786 cpu_manager.go:414] "containerMap: removing container" podUID="b8bf9c7a-9890-4a39-98eb-1a2044daf167" containerName="maas-api" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599923 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8bf9c7a-9890-4a39-98eb-1a2044daf167" containerName="maas-api" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599938 5786 cpu_manager.go:414] "containerMap: removing container" podUID="08f1f1d7-b68a-489c-8914-6525a4079c6a" containerName="maas-api" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599945 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f1f1d7-b68a-489c-8914-6525a4079c6a" containerName="maas-api" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599955 5786 cpu_manager.go:414] "containerMap: removing container" podUID="f10046d1-f25f-4ccd-9139-1f2a864d84e7" containerName="payload-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599963 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10046d1-f25f-4ccd-9139-1f2a864d84e7" containerName="payload-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599973 5786 cpu_manager.go:414] "containerMap: removing container" podUID="978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac" containerName="payload-pre-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599981 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac" containerName="payload-pre-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.599994 5786 cpu_manager.go:414] "containerMap: removing container" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600001 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600014 5786 cpu_manager.go:414] "containerMap: removing container" podUID="705339b0-c4d3-46ad-b55e-919f4113cecc" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600022 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705339b0-c4d3-46ad-b55e-919f4113cecc" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600034 5786 cpu_manager.go:414] "containerMap: removing container" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600042 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600149 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc18916f-4fd9-4086-a1d5-e8036b0d3165" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600163 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="9840601d-5ec0-4a50-8088-64b0ac4fdc4e" containerName="payload-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600173 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="e496528c-9914-4f1f-bb29-f9d10ac5b3b2" containerName="payload-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600184 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec1b4ed-2b7c-4c22-931a-d8b509afd5d4" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600196 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e0983c-143e-4cc0-a647-8a1ac8d18ff2" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600205 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600214 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad79149-b0b3-4a22-a116-8bc3ca56f325" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600222 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54f93ab-904a-4feb-a157-fa470223d756" containerName="main" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600230 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a38ad2e-6b50-457c-874b-b4b7944c63c2" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600238 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5958d8a-1f07-45d1-8410-57494d457efe" containerName="maas-api" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600247 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd9752e-6a9b-470f-ad9d-938bca9a46cc" containerName="payload-pre-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600257 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7102928-5338-4f63-950b-0900b80b07d3" containerName="main" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600266 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="028d9ee5-74e8-448d-b697-d636933c0643" containerName="payload-pre-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600274 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad8d0d6-46c6-4df3-ae85-50ce778aa84c" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600283 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ece2c47-d09b-4355-8b9d-7df3a80de475" containerName="test-curl-7240-7b26ca" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600292 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57ffc7e-b9c6-44e0-a5d3-08df226ee170" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600298 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600307 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="4faf119c-25e3-4f4e-80d7-998f0f46a492" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600318 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8bf9c7a-9890-4a39-98eb-1a2044daf167" containerName="maas-api" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600327 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f1f1d7-b68a-489c-8914-6525a4079c6a" containerName="maas-api" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600336 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10046d1-f25f-4ccd-9139-1f2a864d84e7" containerName="payload-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600345 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="978c1ab7-fbe9-4cfd-a7c6-d5d8f5ca64ac" containerName="payload-pre-processing" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600353 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2502a53-973b-4987-9890-f1172bf9b84d" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600360 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600369 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="705339b0-c4d3-46ad-b55e-919f4113cecc" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600376 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23550ae-bf0a-44d3-960f-1e10c36b518c" containerName="istio-proxy" Nov 04 01:35:11.600422 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.600385 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0ed274-cc86-4ab4-b8fb-db31ee650267" containerName="maas-api" Nov 04 01:35:11.605213 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.605193 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:11.659213 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.659187 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 01:35:11.703791 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.703756 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 01:35:11.703932 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.703817 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4whbk\"" Nov 04 01:35:11.703932 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.703903 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 01:35:11.707936 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.707918 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-566cd577c-zqm98"] Nov 04 01:35:11.727741 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.727715 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-metrics-certs\") pod \"maas-controller-566cd577c-zqm98\" (UID: \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\") " pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:11.727862 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.727749 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-webhook-cert\") pod \"maas-controller-566cd577c-zqm98\" (UID: \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\") " pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:11.727907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.727872 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4vk6\" (UniqueName: \"kubernetes.io/projected/62c29661-e1cb-46ba-b0bc-0600094b1bc5-kube-api-access-p4vk6\") pod \"maas-controller-566cd577c-zqm98\" (UID: \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\") " pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:11.829095 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.829058 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-metrics-certs\") pod \"maas-controller-566cd577c-zqm98\" (UID: \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\") " pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:11.829095 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.829103 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-webhook-cert\") pod \"maas-controller-566cd577c-zqm98\" (UID: \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\") " pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:11.829336 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.829182 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4vk6\" (UniqueName: \"kubernetes.io/projected/62c29661-e1cb-46ba-b0bc-0600094b1bc5-kube-api-access-p4vk6\") pod \"maas-controller-566cd577c-zqm98\" (UID: \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\") " pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:11.831681 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.831653 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-metrics-certs\") pod \"maas-controller-566cd577c-zqm98\" (UID: \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\") " pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:11.831741 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.831701 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-webhook-cert\") pod \"maas-controller-566cd577c-zqm98\" (UID: \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\") " pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:11.885383 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.885315 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4vk6\" (UniqueName: \"kubernetes.io/projected/62c29661-e1cb-46ba-b0bc-0600094b1bc5-kube-api-access-p4vk6\") pod \"maas-controller-566cd577c-zqm98\" (UID: \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\") " pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:11.917301 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:11.917250 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:12.285939 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:12.285858 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-566cd577c-zqm98"] Nov 04 01:35:12.292244 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:35:12.292215 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c29661_e1cb_46ba_b0bc_0600094b1bc5.slice/crio-ad71711a4e0b9c29e9da2c16321b19a4ebc2fc833c75158b9796b96d6232b4d4 WatchSource:0}: Error finding container ad71711a4e0b9c29e9da2c16321b19a4ebc2fc833c75158b9796b96d6232b4d4: Status 404 returned error can't find the container with id ad71711a4e0b9c29e9da2c16321b19a4ebc2fc833c75158b9796b96d6232b4d4 Nov 04 01:35:12.293515 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:12.293498 5786 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:35:13.065033 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:13.064996 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566cd577c-zqm98" event={"ID":"62c29661-e1cb-46ba-b0bc-0600094b1bc5","Type":"ContainerStarted","Data":"ad71711a4e0b9c29e9da2c16321b19a4ebc2fc833c75158b9796b96d6232b4d4"} Nov 04 01:35:14.474948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:14.474873 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:35:15.087323 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:35:15.087289 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:35:18.090181 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:18.090146 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566cd577c-zqm98" event={"ID":"62c29661-e1cb-46ba-b0bc-0600094b1bc5","Type":"ContainerStarted","Data":"4cf4c45013a4352bd1fe7397afd6c2b124799c324617df440862acdf047410f0"} Nov 04 01:35:18.090558 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:18.090279 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:18.220762 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:18.220702 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-566cd577c-zqm98" podStartSLOduration=1.5165935560000001 podStartE2EDuration="7.220690683s" totalImagesPullingTime="5.704097127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:18.218931622 +0000 UTC m=+2263.835277384" watchObservedRunningTime="2026-11-04 01:35:18.220690683 +0000 UTC m=+2263.837036445" Nov 04 01:35:29.097567 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:29.097529 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:32.974541 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:32.974500 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9"] Nov 04 01:35:32.974951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:32.974860 5786 cpu_manager.go:414] "containerMap: removing container" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" Nov 04 01:35:32.974951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:32.974872 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09a5559-a4e6-41f4-9dca-1806e1f61a8e" containerName="istio-proxy" Nov 04 01:35:32.975023 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:32.974970 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="febf58ae-2630-40f9-a463-be0d0c583f2b" containerName="istio-proxy" Nov 04 01:35:32.977984 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:32.977964 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.019961 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.019923 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2434f011-3d55-42fa-8a2d-b07df64862ef-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.020150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.019970 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2434f011-3d55-42fa-8a2d-b07df64862ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.020150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.020052 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.020150 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.020096 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzwz7\" (UniqueName: \"kubernetes.io/projected/2434f011-3d55-42fa-8a2d-b07df64862ef-kube-api-access-kzwz7\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.020268 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.020189 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2434f011-3d55-42fa-8a2d-b07df64862ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.020268 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.020246 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.020332 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.020313 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.020371 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.020337 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.020371 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.020356 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.022399 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.022371 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh"] Nov 04 01:35:33.025791 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.025776 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w"] Nov 04 01:35:33.025905 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.025889 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.029008 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.028988 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.069035 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.069004 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fwrps\"" Nov 04 01:35:33.075437 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.075404 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9"] Nov 04 01:35:33.121484 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121449 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.121692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121492 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.121692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121527 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.121692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121554 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.121692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121591 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.121692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121618 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2434f011-3d55-42fa-8a2d-b07df64862ef-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.121692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121640 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsggl\" (UniqueName: \"kubernetes.io/projected/d802bf36-fe5d-46cf-ab84-208d75861871-kube-api-access-hsggl\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.121692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121667 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.122057 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121699 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d802bf36-fe5d-46cf-ab84-208d75861871-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.122057 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121848 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2434f011-3d55-42fa-8a2d-b07df64862ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.122057 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121936 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.122057 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121946 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.122057 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.121966 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122110 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122161 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543f2b63-4c53-4e48-b7e7-048c472336df-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122222 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122222 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122266 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122319 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122366 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122482 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543f2b63-4c53-4e48-b7e7-048c472336df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122518 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzwz7\" (UniqueName: \"kubernetes.io/projected/2434f011-3d55-42fa-8a2d-b07df64862ef-kube-api-access-kzwz7\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122558 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d802bf36-fe5d-46cf-ab84-208d75861871-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122601 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122607 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d802bf36-fe5d-46cf-ab84-208d75861871-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.122654 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122646 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.123366 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122671 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.123366 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122750 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjq5k\" (UniqueName: \"kubernetes.io/projected/543f2b63-4c53-4e48-b7e7-048c472336df-kube-api-access-xjq5k\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.123366 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122801 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2434f011-3d55-42fa-8a2d-b07df64862ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.123366 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122911 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2434f011-3d55-42fa-8a2d-b07df64862ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.123366 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122946 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543f2b63-4c53-4e48-b7e7-048c472336df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.123366 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.122981 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.125269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.125243 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh"] Nov 04 01:35:33.125595 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.125511 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2434f011-3d55-42fa-8a2d-b07df64862ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.125988 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.125945 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2434f011-3d55-42fa-8a2d-b07df64862ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.127138 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.127090 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w"] Nov 04 01:35:33.178283 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.178250 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2434f011-3d55-42fa-8a2d-b07df64862ef-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.178448 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.178398 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzwz7\" (UniqueName: \"kubernetes.io/projected/2434f011-3d55-42fa-8a2d-b07df64862ef-kube-api-access-kzwz7\") pod \"maas-default-gateway-openshift-default-694db55c87-mjcm9\" (UID: \"2434f011-3d55-42fa-8a2d-b07df64862ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.223497 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223465 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.223497 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223495 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.223725 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223511 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.223725 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223531 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543f2b63-4c53-4e48-b7e7-048c472336df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.223725 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223562 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d802bf36-fe5d-46cf-ab84-208d75861871-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.223725 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223660 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d802bf36-fe5d-46cf-ab84-208d75861871-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.223725 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223700 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.223977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223731 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.223977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223762 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjq5k\" (UniqueName: \"kubernetes.io/projected/543f2b63-4c53-4e48-b7e7-048c472336df-kube-api-access-xjq5k\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.223977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223811 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543f2b63-4c53-4e48-b7e7-048c472336df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.223977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223846 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.223977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223928 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.223977 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223940 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.224341 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223979 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsggl\" (UniqueName: \"kubernetes.io/projected/d802bf36-fe5d-46cf-ab84-208d75861871-kube-api-access-hsggl\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.224341 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.223995 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.224341 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.224009 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.224341 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.224077 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.224341 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.224216 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.224907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.224277 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d802bf36-fe5d-46cf-ab84-208d75861871-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.224907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.224877 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543f2b63-4c53-4e48-b7e7-048c472336df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.224907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.224299 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.225162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.224386 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d802bf36-fe5d-46cf-ab84-208d75861871-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.225162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.224939 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.225162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.224381 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.225162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.224994 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.225162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.225035 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543f2b63-4c53-4e48-b7e7-048c472336df-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.225469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.225192 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.225577 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.225554 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.226202 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.226177 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543f2b63-4c53-4e48-b7e7-048c472336df-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.226347 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.226328 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d802bf36-fe5d-46cf-ab84-208d75861871-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.226593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.226572 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d802bf36-fe5d-46cf-ab84-208d75861871-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.226711 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.226695 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543f2b63-4c53-4e48-b7e7-048c472336df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.280202 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.280174 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d802bf36-fe5d-46cf-ab84-208d75861871-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.280358 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.280334 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjq5k\" (UniqueName: \"kubernetes.io/projected/543f2b63-4c53-4e48-b7e7-048c472336df-kube-api-access-xjq5k\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.280401 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.280334 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsggl\" (UniqueName: \"kubernetes.io/projected/d802bf36-fe5d-46cf-ab84-208d75861871-kube-api-access-hsggl\") pod \"maas-default-gateway-openshift-default-694db55c87-qvl5w\" (UID: \"d802bf36-fe5d-46cf-ab84-208d75861871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.290164 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.290146 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:33.292533 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.292509 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543f2b63-4c53-4e48-b7e7-048c472336df-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-rhjlh\" (UID: \"543f2b63-4c53-4e48-b7e7-048c472336df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.340431 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.340374 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:33.350460 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.350433 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:33.723050 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.723016 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9"] Nov 04 01:35:33.755587 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:35:33.755561 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2434f011_3d55_42fa_8a2d_b07df64862ef.slice/crio-5231a9a86b452865a93889888b7e90cb7050d537b7d32543ceab31964cca4b95 WatchSource:0}: Error finding container 5231a9a86b452865a93889888b7e90cb7050d537b7d32543ceab31964cca4b95: Status 404 returned error can't find the container with id 5231a9a86b452865a93889888b7e90cb7050d537b7d32543ceab31964cca4b95 Nov 04 01:35:33.757609 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.757577 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:33.757707 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.757645 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:33.757707 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.757672 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:33.780110 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.780085 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh"] Nov 04 01:35:33.789382 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:35:33.789353 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543f2b63_4c53_4e48_b7e7_048c472336df.slice/crio-c0264862a43b8d21a607ea91292693b74d92ab442c239e5dcea41250c7a61a72 WatchSource:0}: Error finding container c0264862a43b8d21a607ea91292693b74d92ab442c239e5dcea41250c7a61a72: Status 404 returned error can't find the container with id c0264862a43b8d21a607ea91292693b74d92ab442c239e5dcea41250c7a61a72 Nov 04 01:35:33.791394 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.791361 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:33.791482 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.791423 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:33.791482 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.791450 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:33.813290 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.813266 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w"] Nov 04 01:35:33.824703 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:35:33.824678 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd802bf36_fe5d_46cf_ab84_208d75861871.slice/crio-625d9ec6287ac7f18a18766cab81975ee37edd5f795bfb0777e44482a9a5462f WatchSource:0}: Error finding container 625d9ec6287ac7f18a18766cab81975ee37edd5f795bfb0777e44482a9a5462f: Status 404 returned error can't find the container with id 625d9ec6287ac7f18a18766cab81975ee37edd5f795bfb0777e44482a9a5462f Nov 04 01:35:33.826876 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.826844 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:33.826980 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.826920 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:33.826980 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:33.826967 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:34.163671 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.163633 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" event={"ID":"543f2b63-4c53-4e48-b7e7-048c472336df","Type":"ContainerStarted","Data":"65b60b13add68de09f3205ba1a4f8a0b4a271416717652c2ac44c580bb420144"} Nov 04 01:35:34.164198 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.163696 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" event={"ID":"543f2b63-4c53-4e48-b7e7-048c472336df","Type":"ContainerStarted","Data":"c0264862a43b8d21a607ea91292693b74d92ab442c239e5dcea41250c7a61a72"} Nov 04 01:35:34.166154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.166099 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" event={"ID":"d802bf36-fe5d-46cf-ab84-208d75861871","Type":"ContainerStarted","Data":"2a7db32d72e5504666b826f065a948af53deb62b5fddb9be4fc74edfa26529fa"} Nov 04 01:35:34.166281 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.166167 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" event={"ID":"d802bf36-fe5d-46cf-ab84-208d75861871","Type":"ContainerStarted","Data":"625d9ec6287ac7f18a18766cab81975ee37edd5f795bfb0777e44482a9a5462f"} Nov 04 01:35:34.167991 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.167964 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" event={"ID":"2434f011-3d55-42fa-8a2d-b07df64862ef","Type":"ContainerStarted","Data":"7f50f24755f820f2fa4206051552651adb82130da6794565d527a97558ee65ce"} Nov 04 01:35:34.168098 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.168012 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" event={"ID":"2434f011-3d55-42fa-8a2d-b07df64862ef","Type":"ContainerStarted","Data":"5231a9a86b452865a93889888b7e90cb7050d537b7d32543ceab31964cca4b95"} Nov 04 01:35:34.281896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.281830 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" podStartSLOduration=2.281811978 podStartE2EDuration="2.281811978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:34.279596992 +0000 UTC m=+2279.895942755" watchObservedRunningTime="2026-11-04 01:35:34.281811978 +0000 UTC m=+2279.898157740" Nov 04 01:35:34.292525 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.292484 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:34.293823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.293791 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mjcm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:35:34.293979 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.293854 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" podUID="2434f011-3d55-42fa-8a2d-b07df64862ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 04 01:35:34.341291 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.341256 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:34.342614 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.342586 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rhjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:35:34.342708 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.342648 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" podUID="543f2b63-4c53-4e48-b7e7-048c472336df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 04 01:35:34.355162 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.354500 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:34.359141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.357811 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvl5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:35:34.359141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.357883 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" podUID="d802bf36-fe5d-46cf-ab84-208d75861871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 04 01:35:34.541353 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.541245 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" podStartSLOduration=2.541231401 podStartE2EDuration="2.541231401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:34.405587133 +0000 UTC m=+2280.021932897" watchObservedRunningTime="2026-11-04 01:35:34.541231401 +0000 UTC m=+2280.157577160" Nov 04 01:35:34.541587 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:34.541566 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" podStartSLOduration=2.541561652 podStartE2EDuration="2.541561652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:34.523024878 +0000 UTC m=+2280.139370684" watchObservedRunningTime="2026-11-04 01:35:34.541561652 +0000 UTC m=+2280.157907414" Nov 04 01:35:35.290751 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:35.290713 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mjcm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:35:35.291342 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:35.291311 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" podUID="2434f011-3d55-42fa-8a2d-b07df64862ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 04 01:35:35.340845 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:35.340814 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rhjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:35:35.341077 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:35.341056 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" podUID="543f2b63-4c53-4e48-b7e7-048c472336df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 04 01:35:35.353456 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:35.353427 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvl5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:35:35.353568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:35.353475 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" podUID="d802bf36-fe5d-46cf-ab84-208d75861871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 04 01:35:36.300148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:36.297500 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mjcm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:35:36.300148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:36.297563 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" podUID="2434f011-3d55-42fa-8a2d-b07df64862ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 04 01:35:36.342733 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:36.342697 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rhjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:35:36.342940 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:36.342755 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" podUID="543f2b63-4c53-4e48-b7e7-048c472336df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 04 01:35:36.352419 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:36.352387 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvl5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:35:36.352586 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:36.352435 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" podUID="d802bf36-fe5d-46cf-ab84-208d75861871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 04 01:35:37.305143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:37.295043 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mjcm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:35:37.305143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:37.295110 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" podUID="2434f011-3d55-42fa-8a2d-b07df64862ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 04 01:35:37.343141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:37.342727 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rhjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:35:37.343141 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:37.342791 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" podUID="543f2b63-4c53-4e48-b7e7-048c472336df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 04 01:35:37.352564 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:37.352532 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvl5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:35:37.352718 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:37.352575 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" podUID="d802bf36-fe5d-46cf-ab84-208d75861871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 04 01:35:38.302142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:38.301935 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mjcm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:35:38.302142 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:38.301995 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" podUID="2434f011-3d55-42fa-8a2d-b07df64862ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 04 01:35:38.346153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:38.343095 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rhjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:35:38.346153 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:38.343195 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" podUID="543f2b63-4c53-4e48-b7e7-048c472336df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 04 01:35:38.353456 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:38.353426 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvl5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:35:38.353591 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:38.353478 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" podUID="d802bf36-fe5d-46cf-ab84-208d75861871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 04 01:35:39.294144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:39.293639 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mjcm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:35:39.294144 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:39.293793 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" podUID="2434f011-3d55-42fa-8a2d-b07df64862ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 04 01:35:39.341727 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:39.341673 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rhjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:35:39.341911 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:39.341740 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" podUID="543f2b63-4c53-4e48-b7e7-048c472336df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 04 01:35:39.352948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:39.352904 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvl5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:35:39.353307 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:39.352964 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" podUID="d802bf36-fe5d-46cf-ab84-208d75861871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 04 01:35:40.290791 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:40.290750 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mjcm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:35:40.291000 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:40.290816 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" podUID="2434f011-3d55-42fa-8a2d-b07df64862ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 04 01:35:40.341331 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:40.341293 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rhjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:35:40.341511 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:40.341372 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" podUID="543f2b63-4c53-4e48-b7e7-048c472336df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 04 01:35:40.353419 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:40.353388 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvl5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:35:40.353756 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:40.353441 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" podUID="d802bf36-fe5d-46cf-ab84-208d75861871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 04 01:35:41.290957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:41.290924 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mjcm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:35:41.291148 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:41.290986 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" podUID="2434f011-3d55-42fa-8a2d-b07df64862ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 04 01:35:41.341421 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:41.341389 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rhjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 04 01:35:41.341615 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:41.341444 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" podUID="543f2b63-4c53-4e48-b7e7-048c472336df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 04 01:35:41.353300 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:41.353263 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvl5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 04 01:35:41.353478 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:41.353325 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" podUID="d802bf36-fe5d-46cf-ab84-208d75861871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 04 01:35:42.506308 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:42.506277 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:42.506905 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:42.506887 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:42.507925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:42.507902 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rhjlh" Nov 04 01:35:42.645370 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:42.645340 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:42.646440 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:42.646271 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:42.647159 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:42.647136 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" Nov 04 01:35:42.723229 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:42.723149 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:42.724139 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:42.724095 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:42.724844 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:42.724824 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" Nov 04 01:35:45.140828 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:35:45.140782 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:35:48.095601 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.095562 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688"] Nov 04 01:35:48.100840 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.100815 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.200019 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.199982 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688"] Nov 04 01:35:48.265355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.265318 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47f6c57-b397-4190-b510-bab0db9443a3-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.265355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.265353 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.265574 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.265372 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47f6c57-b397-4190-b510-bab0db9443a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.265574 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.265393 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.265574 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.265482 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.265688 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.265575 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.265688 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.265609 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npgb8\" (UniqueName: \"kubernetes.io/projected/a47f6c57-b397-4190-b510-bab0db9443a3-kube-api-access-npgb8\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.265761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.265707 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.265849 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.265833 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47f6c57-b397-4190-b510-bab0db9443a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.366413 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.366335 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.366572 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.366493 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47f6c57-b397-4190-b510-bab0db9443a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.366572 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.366553 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47f6c57-b397-4190-b510-bab0db9443a3-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.366704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.366578 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.366704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.366606 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47f6c57-b397-4190-b510-bab0db9443a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.366704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.366660 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.366704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.366688 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.366914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.366728 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.366914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.366778 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npgb8\" (UniqueName: \"kubernetes.io/projected/a47f6c57-b397-4190-b510-bab0db9443a3-kube-api-access-npgb8\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.366914 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.366810 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.367078 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.366934 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.367173 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.367105 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.367240 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.367173 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.367294 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.367233 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47f6c57-b397-4190-b510-bab0db9443a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.369037 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.369013 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47f6c57-b397-4190-b510-bab0db9443a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.369252 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.369234 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47f6c57-b397-4190-b510-bab0db9443a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.421931 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.421894 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47f6c57-b397-4190-b510-bab0db9443a3-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.422082 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.422027 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npgb8\" (UniqueName: \"kubernetes.io/projected/a47f6c57-b397-4190-b510-bab0db9443a3-kube-api-access-npgb8\") pod \"maas-default-gateway-openshift-default-694db55c87-ct688\" (UID: \"a47f6c57-b397-4190-b510-bab0db9443a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:48.714019 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:48.713921 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:49.133459 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:49.133420 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688"] Nov 04 01:35:49.138579 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:35:49.138550 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47f6c57_b397_4190_b510_bab0db9443a3.slice/crio-4421c026501b06d5e158cf78c2d60c7881bd51c2b0afef4310d35602708c7a47 WatchSource:0}: Error finding container 4421c026501b06d5e158cf78c2d60c7881bd51c2b0afef4310d35602708c7a47: Status 404 returned error can't find the container with id 4421c026501b06d5e158cf78c2d60c7881bd51c2b0afef4310d35602708c7a47 Nov 04 01:35:49.140912 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:49.140874 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:49.141010 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:49.140961 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:49.141010 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:49.140998 5786 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:49.332057 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:49.332024 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" event={"ID":"a47f6c57-b397-4190-b510-bab0db9443a3","Type":"ContainerStarted","Data":"46c2a21e9859728c2ad167e8e30091ee5ed486b5941ef5839fa41bc296eedeb8"} Nov 04 01:35:49.332251 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:49.332077 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" event={"ID":"a47f6c57-b397-4190-b510-bab0db9443a3","Type":"ContainerStarted","Data":"4421c026501b06d5e158cf78c2d60c7881bd51c2b0afef4310d35602708c7a47"} Nov 04 01:35:49.449262 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:49.449006 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" podStartSLOduration=1.448978831 podStartE2EDuration="1.448978831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:49.445623902 +0000 UTC m=+2295.061969665" watchObservedRunningTime="2026-11-04 01:35:49.448978831 +0000 UTC m=+2295.065324586" Nov 04 01:35:49.714989 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:49.714896 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:50.715848 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:50.715814 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ct688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:35:50.716285 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:50.715882 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" podUID="a47f6c57-b397-4190-b510-bab0db9443a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 04 01:35:51.005996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.005911 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-566cd577c-zqm98"] Nov 04 01:35:51.006170 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.006132 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-566cd577c-zqm98" podUID="62c29661-e1cb-46ba-b0bc-0600094b1bc5" containerName="manager" containerID="cri-o://4cf4c45013a4352bd1fe7397afd6c2b124799c324617df440862acdf047410f0" gracePeriod=30 Nov 04 01:35:51.342244 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.342165 5786 generic.go:356] "Generic (PLEG): container finished" podID="62c29661-e1cb-46ba-b0bc-0600094b1bc5" containerID="4cf4c45013a4352bd1fe7397afd6c2b124799c324617df440862acdf047410f0" exitCode=0 Nov 04 01:35:51.342389 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.342239 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566cd577c-zqm98" event={"ID":"62c29661-e1cb-46ba-b0bc-0600094b1bc5","Type":"ContainerDied","Data":"4cf4c45013a4352bd1fe7397afd6c2b124799c324617df440862acdf047410f0"} Nov 04 01:35:51.714756 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.714722 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ct688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:35:51.714907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.714784 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" podUID="a47f6c57-b397-4190-b510-bab0db9443a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 04 01:35:51.810058 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.810036 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:51.900075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.899992 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-webhook-cert\" (UniqueName: \"kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-webhook-cert\") pod \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\" (UID: \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\") " Nov 04 01:35:51.900075 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.900059 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c29661-e1cb-46ba-b0bc-0600094b1bc5-kube-api-access-p4vk6\" (UniqueName: \"kubernetes.io/projected/62c29661-e1cb-46ba-b0bc-0600094b1bc5-kube-api-access-p4vk6\") pod \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\" (UID: \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\") " Nov 04 01:35:51.900340 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.900079 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-metrics-certs\") pod \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\" (UID: \"62c29661-e1cb-46ba-b0bc-0600094b1bc5\") " Nov 04 01:35:51.902434 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.902376 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-webhook-cert" pod "62c29661-e1cb-46ba-b0bc-0600094b1bc5" (UID: "62c29661-e1cb-46ba-b0bc-0600094b1bc5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:51.902568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.902494 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-metrics-certs" pod "62c29661-e1cb-46ba-b0bc-0600094b1bc5" (UID: "62c29661-e1cb-46ba-b0bc-0600094b1bc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:51.902568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:51.902510 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c29661-e1cb-46ba-b0bc-0600094b1bc5-kube-api-access-p4vk6" pod "62c29661-e1cb-46ba-b0bc-0600094b1bc5" (UID: "62c29661-e1cb-46ba-b0bc-0600094b1bc5"). InnerVolumeSpecName "kube-api-access-p4vk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:52.001157 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:52.001100 5786 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-webhook-cert\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:52.001316 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:52.001159 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4vk6\" (UniqueName: \"kubernetes.io/projected/62c29661-e1cb-46ba-b0bc-0600094b1bc5-kube-api-access-p4vk6\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:52.001316 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:52.001176 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c29661-e1cb-46ba-b0bc-0600094b1bc5-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:52.348741 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:52.348710 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566cd577c-zqm98" event={"ID":"62c29661-e1cb-46ba-b0bc-0600094b1bc5","Type":"ContainerDied","Data":"ad71711a4e0b9c29e9da2c16321b19a4ebc2fc833c75158b9796b96d6232b4d4"} Nov 04 01:35:52.348913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:52.348749 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566cd577c-zqm98" Nov 04 01:35:52.348913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:52.348798 5786 scope.go:122] "RemoveContainer" containerID="4cf4c45013a4352bd1fe7397afd6c2b124799c324617df440862acdf047410f0" Nov 04 01:35:52.510257 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:52.510226 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-566cd577c-zqm98"] Nov 04 01:35:52.557891 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:52.557859 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-566cd577c-zqm98"] Nov 04 01:35:52.714713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:52.714632 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ct688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:35:52.714713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:52.714699 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" podUID="a47f6c57-b397-4190-b510-bab0db9443a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 04 01:35:53.714409 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:53.714373 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ct688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:35:53.714833 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:53.714462 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" podUID="a47f6c57-b397-4190-b510-bab0db9443a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 04 01:35:54.018529 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.018451 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-566cd577c-j4xcj"] Nov 04 01:35:54.018858 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.018845 5786 cpu_manager.go:414] "containerMap: removing container" podUID="62c29661-e1cb-46ba-b0bc-0600094b1bc5" containerName="manager" Nov 04 01:35:54.018906 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.018859 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c29661-e1cb-46ba-b0bc-0600094b1bc5" containerName="manager" Nov 04 01:35:54.018941 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.018932 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c29661-e1cb-46ba-b0bc-0600094b1bc5" containerName="manager" Nov 04 01:35:54.022379 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.022360 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:35:54.074110 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.074076 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 01:35:54.121369 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.121338 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 01:35:54.121566 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.121353 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4whbk\"" Nov 04 01:35:54.121566 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.121522 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 01:35:54.121566 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.121520 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p2ct\" (UniqueName: \"kubernetes.io/projected/00f9fdff-f532-47b4-ae77-6cf7e35aa000-kube-api-access-8p2ct\") pod \"maas-controller-566cd577c-j4xcj\" (UID: \"00f9fdff-f532-47b4-ae77-6cf7e35aa000\") " pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:35:54.121699 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.121597 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00f9fdff-f532-47b4-ae77-6cf7e35aa000-webhook-cert\") pod \"maas-controller-566cd577c-j4xcj\" (UID: \"00f9fdff-f532-47b4-ae77-6cf7e35aa000\") " pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:35:54.121699 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.121678 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f9fdff-f532-47b4-ae77-6cf7e35aa000-metrics-certs\") pod \"maas-controller-566cd577c-j4xcj\" (UID: \"00f9fdff-f532-47b4-ae77-6cf7e35aa000\") " pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:35:54.122987 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.122967 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-566cd577c-j4xcj"] Nov 04 01:35:54.222324 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.222286 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p2ct\" (UniqueName: \"kubernetes.io/projected/00f9fdff-f532-47b4-ae77-6cf7e35aa000-kube-api-access-8p2ct\") pod \"maas-controller-566cd577c-j4xcj\" (UID: \"00f9fdff-f532-47b4-ae77-6cf7e35aa000\") " pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:35:54.222498 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.222346 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00f9fdff-f532-47b4-ae77-6cf7e35aa000-webhook-cert\") pod \"maas-controller-566cd577c-j4xcj\" (UID: \"00f9fdff-f532-47b4-ae77-6cf7e35aa000\") " pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:35:54.222549 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.222517 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f9fdff-f532-47b4-ae77-6cf7e35aa000-metrics-certs\") pod \"maas-controller-566cd577c-j4xcj\" (UID: \"00f9fdff-f532-47b4-ae77-6cf7e35aa000\") " pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:35:54.224787 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.224766 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00f9fdff-f532-47b4-ae77-6cf7e35aa000-webhook-cert\") pod \"maas-controller-566cd577c-j4xcj\" (UID: \"00f9fdff-f532-47b4-ae77-6cf7e35aa000\") " pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:35:54.224901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.224816 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00f9fdff-f532-47b4-ae77-6cf7e35aa000-metrics-certs\") pod \"maas-controller-566cd577c-j4xcj\" (UID: \"00f9fdff-f532-47b4-ae77-6cf7e35aa000\") " pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:35:54.287943 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.287810 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p2ct\" (UniqueName: \"kubernetes.io/projected/00f9fdff-f532-47b4-ae77-6cf7e35aa000-kube-api-access-8p2ct\") pod \"maas-controller-566cd577c-j4xcj\" (UID: \"00f9fdff-f532-47b4-ae77-6cf7e35aa000\") " pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:35:54.334886 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.334842 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:35:54.510540 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.510499 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c29661-e1cb-46ba-b0bc-0600094b1bc5" path="/var/lib/kubelet/pods/62c29661-e1cb-46ba-b0bc-0600094b1bc5/volumes" Nov 04 01:35:54.715241 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.715201 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ct688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:35:54.715625 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.715263 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" podUID="a47f6c57-b397-4190-b510-bab0db9443a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 04 01:35:54.729229 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:54.729200 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-566cd577c-j4xcj"] Nov 04 01:35:54.735151 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:35:54.735105 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f9fdff_f532_47b4_ae77_6cf7e35aa000.slice/crio-af3f5f2a824b1e578865e456866a9638f7df2568c5406d5f18b36efa3cfd581e WatchSource:0}: Error finding container af3f5f2a824b1e578865e456866a9638f7df2568c5406d5f18b36efa3cfd581e: Status 404 returned error can't find the container with id af3f5f2a824b1e578865e456866a9638f7df2568c5406d5f18b36efa3cfd581e Nov 04 01:35:55.366714 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:55.366680 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566cd577c-j4xcj" event={"ID":"00f9fdff-f532-47b4-ae77-6cf7e35aa000","Type":"ContainerStarted","Data":"af3f5f2a824b1e578865e456866a9638f7df2568c5406d5f18b36efa3cfd581e"} Nov 04 01:35:55.714926 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:55.714886 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ct688 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:35:55.715084 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:55.714956 5786 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" podUID="a47f6c57-b397-4190-b510-bab0db9443a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 04 01:35:56.374232 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:56.374194 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566cd577c-j4xcj" event={"ID":"00f9fdff-f532-47b4-ae77-6cf7e35aa000","Type":"ContainerStarted","Data":"73bec040c8276c6e6e9013cb67c429f0abac342ac85e5e954c26bcee6f8a4159"} Nov 04 01:35:56.374683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:56.374304 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:35:56.487970 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:56.487913 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-566cd577c-j4xcj" podStartSLOduration=2.600585244 podStartE2EDuration="3.487898198s" totalImagesPullingTime="887.312954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:56.486590371 +0000 UTC m=+2302.102936113" watchObservedRunningTime="2026-11-04 01:35:56.487898198 +0000 UTC m=+2302.104243960" Nov 04 01:35:57.412284 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:57.412253 5786 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:57.412724 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:57.412555 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:35:57.413322 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:35:57.413306 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" Nov 04 01:36:07.383241 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:36:07.383210 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-566cd577c-j4xcj" Nov 04 01:36:15.088159 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:36:15.088088 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:36:37.456094 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:36:37.456061 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:36:45.087040 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:36:45.086957 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:36:58.867946 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:36:58.867900 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:37:09.473845 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:37:09.473811 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:37:15.086488 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:37:15.086445 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:37:34.268390 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:37:34.268355 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:37:34.820622 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:37:34.820587 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:37:34.883506 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:37:34.883474 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:37:45.087359 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:37:45.087311 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:37:47.370211 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:37:47.370178 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:38:15.086632 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:38:15.086546 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:38:18.692113 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:38:18.692071 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:38:29.755165 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:38:29.755113 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:38:39.960962 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:38:39.960928 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:38:45.086657 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:38:45.086619 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:39:04.257137 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:39:04.257092 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:39:14.370884 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:39:14.370845 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:39:15.087515 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:39:15.087478 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:39:35.570242 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:39:35.570206 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:39:45.161109 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:39:45.160933 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:39:48.877409 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:39:48.877368 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:40:11.266828 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:40:11.266793 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:40:13.695466 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:40:13.695410 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mjcm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:40:13.696155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:40:13.696093 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mjcm9" podUID="2434f011-3d55-42fa-8a2d-b07df64862ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:40:13.772319 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:40:13.772278 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvl5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:40:13.772656 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:40:13.772610 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvl5w" podUID="d802bf36-fe5d-46cf-ab84-208d75861871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:40:15.087439 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:40:15.087401 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:40:29.857380 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:40:29.857348 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:40:42.980231 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:40:42.980191 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:40:45.198457 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:40:45.198403 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:41:15.087315 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:41:15.087228 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:41:17.178059 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:41:17.178026 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:41:40.277961 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:41:40.277927 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:41:43.441685 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:41:43.441633 5786 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ct688 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:41:43.442259 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:41:43.441724 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ct688" podUID="a47f6c57-b397-4190-b510-bab0db9443a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:41:45.086911 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:41:45.086872 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:41:51.369832 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:41:51.369794 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:42:15.087264 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:42:15.087225 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:42:26.345700 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.345667 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z"] Nov 04 01:42:26.349602 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.349582 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" Nov 04 01:42:26.390558 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.390513 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621d0fc4-73a7-4959-bb9d-edc064405a71-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z\" (UID: \"621d0fc4-73a7-4959-bb9d-edc064405a71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" Nov 04 01:42:26.390781 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.390691 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqdgw\" (UniqueName: \"kubernetes.io/projected/621d0fc4-73a7-4959-bb9d-edc064405a71-kube-api-access-mqdgw\") pod \"payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z\" (UID: \"621d0fc4-73a7-4959-bb9d-edc064405a71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" Nov 04 01:42:26.401720 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.401681 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f9348a\"" Nov 04 01:42:26.403414 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.403384 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z"] Nov 04 01:42:26.443515 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.443475 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-f4596f44-dockercfg-vk2vb\"" Nov 04 01:42:26.489882 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.489849 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l"] Nov 04 01:42:26.491822 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.491789 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621d0fc4-73a7-4959-bb9d-edc064405a71-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z\" (UID: \"621d0fc4-73a7-4959-bb9d-edc064405a71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" Nov 04 01:42:26.491943 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.491913 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqdgw\" (UniqueName: \"kubernetes.io/projected/621d0fc4-73a7-4959-bb9d-edc064405a71-kube-api-access-mqdgw\") pod \"payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z\" (UID: \"621d0fc4-73a7-4959-bb9d-edc064405a71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" Nov 04 01:42:26.492560 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.492539 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621d0fc4-73a7-4959-bb9d-edc064405a71-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z\" (UID: \"621d0fc4-73a7-4959-bb9d-edc064405a71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" Nov 04 01:42:26.493495 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.493481 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" Nov 04 01:42:26.547470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.547437 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l"] Nov 04 01:42:26.550242 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.550212 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqdgw\" (UniqueName: \"kubernetes.io/projected/621d0fc4-73a7-4959-bb9d-edc064405a71-kube-api-access-mqdgw\") pod \"payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z\" (UID: \"621d0fc4-73a7-4959-bb9d-edc064405a71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" Nov 04 01:42:26.592778 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.592748 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpkqx\" (UniqueName: \"kubernetes.io/projected/e563c25f-bd6b-4a0d-a222-d874f9e8903f-kube-api-access-vpkqx\") pod \"payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l\" (UID: \"e563c25f-bd6b-4a0d-a222-d874f9e8903f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" Nov 04 01:42:26.592969 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.592788 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e563c25f-bd6b-4a0d-a222-d874f9e8903f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l\" (UID: \"e563c25f-bd6b-4a0d-a222-d874f9e8903f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" Nov 04 01:42:26.661915 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.661832 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" Nov 04 01:42:26.693751 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.693715 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpkqx\" (UniqueName: \"kubernetes.io/projected/e563c25f-bd6b-4a0d-a222-d874f9e8903f-kube-api-access-vpkqx\") pod \"payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l\" (UID: \"e563c25f-bd6b-4a0d-a222-d874f9e8903f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" Nov 04 01:42:26.693896 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.693760 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e563c25f-bd6b-4a0d-a222-d874f9e8903f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l\" (UID: \"e563c25f-bd6b-4a0d-a222-d874f9e8903f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" Nov 04 01:42:26.694476 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.694454 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e563c25f-bd6b-4a0d-a222-d874f9e8903f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l\" (UID: \"e563c25f-bd6b-4a0d-a222-d874f9e8903f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" Nov 04 01:42:26.748394 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.748365 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpkqx\" (UniqueName: \"kubernetes.io/projected/e563c25f-bd6b-4a0d-a222-d874f9e8903f-kube-api-access-vpkqx\") pod \"payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l\" (UID: \"e563c25f-bd6b-4a0d-a222-d874f9e8903f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" Nov 04 01:42:26.806140 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:26.806099 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" Nov 04 01:42:27.028731 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:27.028651 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z"] Nov 04 01:42:27.034965 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:42:27.034940 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621d0fc4_73a7_4959_bb9d_edc064405a71.slice/crio-039758732ef3934ce133d5fcbe7d4eb8a12ce7a37ad4d8430eb22715c8b85324 WatchSource:0}: Error finding container 039758732ef3934ce133d5fcbe7d4eb8a12ce7a37ad4d8430eb22715c8b85324: Status 404 returned error can't find the container with id 039758732ef3934ce133d5fcbe7d4eb8a12ce7a37ad4d8430eb22715c8b85324 Nov 04 01:42:27.171856 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:27.171821 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l"] Nov 04 01:42:27.180788 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:42:27.180755 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode563c25f_bd6b_4a0d_a222_d874f9e8903f.slice/crio-2b18fc1b9f4a5b46fed0ddb9e6f03b4272f20aead4497ffb0c42218b2032dcbe WatchSource:0}: Error finding container 2b18fc1b9f4a5b46fed0ddb9e6f03b4272f20aead4497ffb0c42218b2032dcbe: Status 404 returned error can't find the container with id 2b18fc1b9f4a5b46fed0ddb9e6f03b4272f20aead4497ffb0c42218b2032dcbe Nov 04 01:42:27.465602 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:27.465566 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" event={"ID":"e563c25f-bd6b-4a0d-a222-d874f9e8903f","Type":"ContainerStarted","Data":"386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba"} Nov 04 01:42:27.466091 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:27.465622 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" event={"ID":"e563c25f-bd6b-4a0d-a222-d874f9e8903f","Type":"ContainerStarted","Data":"2b18fc1b9f4a5b46fed0ddb9e6f03b4272f20aead4497ffb0c42218b2032dcbe"} Nov 04 01:42:27.466091 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:27.465656 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" Nov 04 01:42:27.467105 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:27.467078 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" event={"ID":"621d0fc4-73a7-4959-bb9d-edc064405a71","Type":"ContainerStarted","Data":"ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03"} Nov 04 01:42:27.467219 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:27.467140 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" event={"ID":"621d0fc4-73a7-4959-bb9d-edc064405a71","Type":"ContainerStarted","Data":"039758732ef3934ce133d5fcbe7d4eb8a12ce7a37ad4d8430eb22715c8b85324"} Nov 04 01:42:27.467219 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:27.467176 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" Nov 04 01:42:27.575643 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:27.575587 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" podStartSLOduration=1.575572269 podStartE2EDuration="1.575572269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:42:27.574436793 +0000 UTC m=+2693.190782538" watchObservedRunningTime="2026-11-04 01:42:27.575572269 +0000 UTC m=+2693.191918031" Nov 04 01:42:27.682783 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:27.682722 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" podStartSLOduration=1.6827093880000001 podStartE2EDuration="1.682709388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:42:27.682286952 +0000 UTC m=+2693.298632718" watchObservedRunningTime="2026-11-04 01:42:27.682709388 +0000 UTC m=+2693.299055142" Nov 04 01:42:28.515753 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.515723 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9"] Nov 04 01:42:28.519245 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.519229 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:28.613693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.613659 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cd807e-82b9-4078-8f42-78b109ed0cf9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:28.613879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.613718 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:28.613879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.613772 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkdqv\" (UniqueName: \"kubernetes.io/projected/f3cd807e-82b9-4078-8f42-78b109ed0cf9-kube-api-access-zkdqv\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:28.613879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.613856 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:28.617104 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.617086 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-f9348a-serving-cert\"" Nov 04 01:42:28.618692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.618673 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9"] Nov 04 01:42:28.714857 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.714822 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:28.715032 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.714896 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cd807e-82b9-4078-8f42-78b109ed0cf9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:28.715032 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.714926 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:28.715032 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.714952 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkdqv\" (UniqueName: \"kubernetes.io/projected/f3cd807e-82b9-4078-8f42-78b109ed0cf9-kube-api-access-zkdqv\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:28.715032 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:42:28.714966 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-serving-cert: secret "maas-api-e2e-worker-single-main-f9348a-serving-cert" not found Nov 04 01:42:28.715241 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:42:28.715050 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-maas-api-tls podName:f3cd807e-82b9-4078-8f42-78b109ed0cf9 nodeName:}" failed. No retries permitted until 2026-11-04 01:42:29.215029054 +0000 UTC m=+2694.831374809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-maas-api-tls") pod "maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" (UID: "f3cd807e-82b9-4078-8f42-78b109ed0cf9") : secret "maas-api-e2e-worker-single-main-f9348a-serving-cert" not found Nov 04 01:42:28.715594 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.715572 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cd807e-82b9-4078-8f42-78b109ed0cf9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:28.717381 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.717364 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:28.774534 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:28.774353 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkdqv\" (UniqueName: \"kubernetes.io/projected/f3cd807e-82b9-4078-8f42-78b109ed0cf9-kube-api-access-zkdqv\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:29.218001 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:29.217965 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:29.220461 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:29.220427 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:29.432486 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:29.432448 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:29.801017 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:29.800940 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9"] Nov 04 01:42:29.806988 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:42:29.806953 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cd807e_82b9_4078_8f42_78b109ed0cf9.slice/crio-9f13fd77b265e7f9bd471b605ddf8c9610cc84a2a3b86c64585516c4f32fa7ce WatchSource:0}: Error finding container 9f13fd77b265e7f9bd471b605ddf8c9610cc84a2a3b86c64585516c4f32fa7ce: Status 404 returned error can't find the container with id 9f13fd77b265e7f9bd471b605ddf8c9610cc84a2a3b86c64585516c4f32fa7ce Nov 04 01:42:29.808472 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:29.808450 5786 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:42:30.484660 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:30.484623 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" event={"ID":"f3cd807e-82b9-4078-8f42-78b109ed0cf9","Type":"ContainerStarted","Data":"9f13fd77b265e7f9bd471b605ddf8c9610cc84a2a3b86c64585516c4f32fa7ce"} Nov 04 01:42:31.491680 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:31.491647 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" event={"ID":"f3cd807e-82b9-4078-8f42-78b109ed0cf9","Type":"ContainerStarted","Data":"d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c"} Nov 04 01:42:31.492234 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:31.491728 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:31.603559 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:31.603506 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" podStartSLOduration=2.6952223440000003 podStartE2EDuration="3.603491529s" totalImagesPullingTime="908.269185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:42:31.603342442 +0000 UTC m=+2697.219688204" watchObservedRunningTime="2026-11-04 01:42:31.603491529 +0000 UTC m=+2697.219837289" Nov 04 01:42:34.856265 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:34.856238 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:42:34.917110 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:34.917072 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:42:36.854101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:36.854069 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:42:37.501777 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:37.501747 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:42:38.475278 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:38.475245 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" Nov 04 01:42:38.475677 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:38.475316 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" Nov 04 01:42:45.087232 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:42:45.087154 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:42:56.763420 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:42:56.763326 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:43:15.087292 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:43:15.087252 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:43:26.678497 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:43:26.678461 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:43:45.087624 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:43:45.087587 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:43:55.880250 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:43:55.880217 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:44:04.370804 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:04.370763 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l"] Nov 04 01:44:04.481734 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:04.481693 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z"] Nov 04 01:44:07.904201 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:07.904164 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9"] Nov 04 01:44:07.904629 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:07.904400 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" podUID="f3cd807e-82b9-4078-8f42-78b109ed0cf9" containerName="maas-api" containerID="cri-o://d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c" gracePeriod=30 Nov 04 01:44:09.156236 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.156206 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:44:09.268168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.267924 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-maas-api-tls\") pod \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " Nov 04 01:44:09.268168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.267989 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-metrics-certs\") pod \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " Nov 04 01:44:09.268168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.268153 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cd807e-82b9-4078-8f42-78b109ed0cf9-kube-api-access-zkdqv\" (UniqueName: \"kubernetes.io/projected/f3cd807e-82b9-4078-8f42-78b109ed0cf9-kube-api-access-zkdqv\") pod \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " Nov 04 01:44:09.268484 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.268190 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cd807e-82b9-4078-8f42-78b109ed0cf9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cd807e-82b9-4078-8f42-78b109ed0cf9-openshift-service-ca\") pod \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\" (UID: \"f3cd807e-82b9-4078-8f42-78b109ed0cf9\") " Nov 04 01:44:09.268814 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.268774 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cd807e-82b9-4078-8f42-78b109ed0cf9-openshift-service-ca" pod "f3cd807e-82b9-4078-8f42-78b109ed0cf9" (UID: "f3cd807e-82b9-4078-8f42-78b109ed0cf9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:44:09.270693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.270666 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-metrics-certs" pod "f3cd807e-82b9-4078-8f42-78b109ed0cf9" (UID: "f3cd807e-82b9-4078-8f42-78b109ed0cf9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:44:09.270877 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.270852 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-maas-api-tls" pod "f3cd807e-82b9-4078-8f42-78b109ed0cf9" (UID: "f3cd807e-82b9-4078-8f42-78b109ed0cf9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:44:09.271301 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.271273 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cd807e-82b9-4078-8f42-78b109ed0cf9-kube-api-access-zkdqv" pod "f3cd807e-82b9-4078-8f42-78b109ed0cf9" (UID: "f3cd807e-82b9-4078-8f42-78b109ed0cf9"). InnerVolumeSpecName "kube-api-access-zkdqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:09.369823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.369784 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkdqv\" (UniqueName: \"kubernetes.io/projected/f3cd807e-82b9-4078-8f42-78b109ed0cf9-kube-api-access-zkdqv\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:09.369823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.369818 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cd807e-82b9-4078-8f42-78b109ed0cf9-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:09.369823 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.369830 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:09.370067 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.369839 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3cd807e-82b9-4078-8f42-78b109ed0cf9-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:09.371547 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.371516 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" podUID="e563c25f-bd6b-4a0d-a222-d874f9e8903f" containerName="payload-processing" containerID="cri-o://386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba" gracePeriod=25 Nov 04 01:44:09.483040 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.482981 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" podUID="621d0fc4-73a7-4959-bb9d-edc064405a71" containerName="payload-pre-processing" containerID="cri-o://ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03" gracePeriod=25 Nov 04 01:44:09.952520 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.952482 5786 generic.go:356] "Generic (PLEG): container finished" podID="f3cd807e-82b9-4078-8f42-78b109ed0cf9" containerID="d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c" exitCode=0 Nov 04 01:44:09.952737 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.952547 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" Nov 04 01:44:09.952737 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.952563 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" event={"ID":"f3cd807e-82b9-4078-8f42-78b109ed0cf9","Type":"ContainerDied","Data":"d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c"} Nov 04 01:44:09.952737 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.952637 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9" event={"ID":"f3cd807e-82b9-4078-8f42-78b109ed0cf9","Type":"ContainerDied","Data":"9f13fd77b265e7f9bd471b605ddf8c9610cc84a2a3b86c64585516c4f32fa7ce"} Nov 04 01:44:09.952737 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.952662 5786 scope.go:122] "RemoveContainer" containerID="d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c" Nov 04 01:44:09.962835 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.962818 5786 scope.go:122] "RemoveContainer" containerID="d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c" Nov 04 01:44:09.963197 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:44:09.963164 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c\": container with ID starting with d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c not found: ID does not exist" containerID="d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c" Nov 04 01:44:09.963308 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:09.963205 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c"} err="failed to get container status \"d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c\": rpc error: code = NotFound desc = could not find container \"d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c\": container with ID starting with d6599ad157f9010300a28e5236471098545acf120403b15afda7978fc287a07c not found: ID does not exist" Nov 04 01:44:10.118245 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:10.118204 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9"] Nov 04 01:44:10.166458 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:10.166414 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f9348a-755b987475-mnzm9"] Nov 04 01:44:10.504583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:10.504553 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cd807e-82b9-4078-8f42-78b109ed0cf9" path="/var/lib/kubelet/pods/f3cd807e-82b9-4078-8f42-78b109ed0cf9/volumes" Nov 04 01:44:12.991088 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:12.991044 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:44:15.086923 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:44:15.086836 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:44:18.473181 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:18.473145 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" podUID="e563c25f-bd6b-4a0d-a222-d874f9e8903f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:44:18.473669 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:18.473156 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" podUID="621d0fc4-73a7-4959-bb9d-edc064405a71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:44:28.472927 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:28.472890 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" podUID="621d0fc4-73a7-4959-bb9d-edc064405a71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:44:28.473362 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:28.472897 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" podUID="e563c25f-bd6b-4a0d-a222-d874f9e8903f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:44:29.623038 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.623008 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" Nov 04 01:44:29.659945 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.659912 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e563c25f-bd6b-4a0d-a222-d874f9e8903f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e563c25f-bd6b-4a0d-a222-d874f9e8903f-plugins-config-volume\") pod \"e563c25f-bd6b-4a0d-a222-d874f9e8903f\" (UID: \"e563c25f-bd6b-4a0d-a222-d874f9e8903f\") " Nov 04 01:44:29.660085 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.660000 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e563c25f-bd6b-4a0d-a222-d874f9e8903f-kube-api-access-vpkqx\" (UniqueName: \"kubernetes.io/projected/e563c25f-bd6b-4a0d-a222-d874f9e8903f-kube-api-access-vpkqx\") pod \"e563c25f-bd6b-4a0d-a222-d874f9e8903f\" (UID: \"e563c25f-bd6b-4a0d-a222-d874f9e8903f\") " Nov 04 01:44:29.660297 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.660276 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e563c25f-bd6b-4a0d-a222-d874f9e8903f-plugins-config-volume" pod "e563c25f-bd6b-4a0d-a222-d874f9e8903f" (UID: "e563c25f-bd6b-4a0d-a222-d874f9e8903f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:44:29.660469 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.660449 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e563c25f-bd6b-4a0d-a222-d874f9e8903f-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:29.662356 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.662332 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e563c25f-bd6b-4a0d-a222-d874f9e8903f-kube-api-access-vpkqx" pod "e563c25f-bd6b-4a0d-a222-d874f9e8903f" (UID: "e563c25f-bd6b-4a0d-a222-d874f9e8903f"). InnerVolumeSpecName "kube-api-access-vpkqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:29.713427 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.713403 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" Nov 04 01:44:29.761420 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.761388 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621d0fc4-73a7-4959-bb9d-edc064405a71-kube-api-access-mqdgw\" (UniqueName: \"kubernetes.io/projected/621d0fc4-73a7-4959-bb9d-edc064405a71-kube-api-access-mqdgw\") pod \"621d0fc4-73a7-4959-bb9d-edc064405a71\" (UID: \"621d0fc4-73a7-4959-bb9d-edc064405a71\") " Nov 04 01:44:29.761592 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.761445 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621d0fc4-73a7-4959-bb9d-edc064405a71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621d0fc4-73a7-4959-bb9d-edc064405a71-plugins-config-volume\") pod \"621d0fc4-73a7-4959-bb9d-edc064405a71\" (UID: \"621d0fc4-73a7-4959-bb9d-edc064405a71\") " Nov 04 01:44:29.761641 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.761628 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpkqx\" (UniqueName: \"kubernetes.io/projected/e563c25f-bd6b-4a0d-a222-d874f9e8903f-kube-api-access-vpkqx\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:29.761918 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.761888 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621d0fc4-73a7-4959-bb9d-edc064405a71-plugins-config-volume" pod "621d0fc4-73a7-4959-bb9d-edc064405a71" (UID: "621d0fc4-73a7-4959-bb9d-edc064405a71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:44:29.763470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.763449 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621d0fc4-73a7-4959-bb9d-edc064405a71-kube-api-access-mqdgw" pod "621d0fc4-73a7-4959-bb9d-edc064405a71" (UID: "621d0fc4-73a7-4959-bb9d-edc064405a71"). InnerVolumeSpecName "kube-api-access-mqdgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:29.862671 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.862639 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqdgw\" (UniqueName: \"kubernetes.io/projected/621d0fc4-73a7-4959-bb9d-edc064405a71-kube-api-access-mqdgw\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:29.862671 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:29.862666 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621d0fc4-73a7-4959-bb9d-edc064405a71-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:30.070532 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.070443 5786 generic.go:356] "Generic (PLEG): container finished" podID="621d0fc4-73a7-4959-bb9d-edc064405a71" containerID="ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03" exitCode=0 Nov 04 01:44:30.070532 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.070489 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" Nov 04 01:44:30.070757 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.070532 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" event={"ID":"621d0fc4-73a7-4959-bb9d-edc064405a71","Type":"ContainerDied","Data":"ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03"} Nov 04 01:44:30.070757 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.070590 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z" event={"ID":"621d0fc4-73a7-4959-bb9d-edc064405a71","Type":"ContainerDied","Data":"039758732ef3934ce133d5fcbe7d4eb8a12ce7a37ad4d8430eb22715c8b85324"} Nov 04 01:44:30.070757 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.070619 5786 scope.go:122] "RemoveContainer" containerID="ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03" Nov 04 01:44:30.071872 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.071852 5786 generic.go:356] "Generic (PLEG): container finished" podID="e563c25f-bd6b-4a0d-a222-d874f9e8903f" containerID="386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba" exitCode=0 Nov 04 01:44:30.071957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.071901 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" event={"ID":"e563c25f-bd6b-4a0d-a222-d874f9e8903f","Type":"ContainerDied","Data":"386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba"} Nov 04 01:44:30.071957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.071915 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" Nov 04 01:44:30.071957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.071936 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l" event={"ID":"e563c25f-bd6b-4a0d-a222-d874f9e8903f","Type":"ContainerDied","Data":"2b18fc1b9f4a5b46fed0ddb9e6f03b4272f20aead4497ffb0c42218b2032dcbe"} Nov 04 01:44:30.080562 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.080488 5786 scope.go:122] "RemoveContainer" containerID="ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03" Nov 04 01:44:30.080823 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:44:30.080798 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03\": container with ID starting with ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03 not found: ID does not exist" containerID="ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03" Nov 04 01:44:30.080902 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.080841 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03"} err="failed to get container status \"ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03\": rpc error: code = NotFound desc = could not find container \"ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03\": container with ID starting with ec9cc2a18e770fda9a58fd4bbb6e1c0f946424cc25fb769af2f58c78d84eed03 not found: ID does not exist" Nov 04 01:44:30.080902 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.080863 5786 scope.go:122] "RemoveContainer" containerID="386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba" Nov 04 01:44:30.088463 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.088443 5786 scope.go:122] "RemoveContainer" containerID="386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba" Nov 04 01:44:30.088785 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:44:30.088765 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba\": container with ID starting with 386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba not found: ID does not exist" containerID="386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba" Nov 04 01:44:30.088854 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.088789 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba"} err="failed to get container status \"386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba\": rpc error: code = NotFound desc = could not find container \"386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba\": container with ID starting with 386ed8207f1bf756c4d9cc72c26ad14ffc8873b58cbc9f6376074e43831b8cba not found: ID does not exist" Nov 04 01:44:30.227129 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.227094 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l"] Nov 04 01:44:30.274996 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.274964 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f9348a-57dfd4849nzg5l"] Nov 04 01:44:30.380323 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.380287 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z"] Nov 04 01:44:30.383992 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.383965 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f9348a-78c599767z"] Nov 04 01:44:30.503619 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.503586 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621d0fc4-73a7-4959-bb9d-edc064405a71" path="/var/lib/kubelet/pods/621d0fc4-73a7-4959-bb9d-edc064405a71/volumes" Nov 04 01:44:30.503967 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:44:30.503954 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e563c25f-bd6b-4a0d-a222-d874f9e8903f" path="/var/lib/kubelet/pods/e563c25f-bd6b-4a0d-a222-d874f9e8903f/volumes" Nov 04 01:44:45.086487 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:44:45.086447 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:45:15.087957 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:45:15.087917 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:45:45.086673 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:45:45.086594 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:46:02.360951 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:02.360915 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:46:15.086867 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:46:15.086831 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:46:19.563021 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:19.562985 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:46:29.251058 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:29.251022 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:46:45.086969 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:46:45.086932 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:46:48.673251 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:48.673219 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:46:56.301960 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.301916 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd"] Nov 04 01:46:56.302395 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.302385 5786 cpu_manager.go:414] "containerMap: removing container" podUID="621d0fc4-73a7-4959-bb9d-edc064405a71" containerName="payload-pre-processing" Nov 04 01:46:56.302455 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.302397 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621d0fc4-73a7-4959-bb9d-edc064405a71" containerName="payload-pre-processing" Nov 04 01:46:56.302455 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.302420 5786 cpu_manager.go:414] "containerMap: removing container" podUID="f3cd807e-82b9-4078-8f42-78b109ed0cf9" containerName="maas-api" Nov 04 01:46:56.302455 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.302425 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cd807e-82b9-4078-8f42-78b109ed0cf9" containerName="maas-api" Nov 04 01:46:56.302455 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.302436 5786 cpu_manager.go:414] "containerMap: removing container" podUID="e563c25f-bd6b-4a0d-a222-d874f9e8903f" containerName="payload-processing" Nov 04 01:46:56.302455 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.302441 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e563c25f-bd6b-4a0d-a222-d874f9e8903f" containerName="payload-processing" Nov 04 01:46:56.302610 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.302514 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="621d0fc4-73a7-4959-bb9d-edc064405a71" containerName="payload-pre-processing" Nov 04 01:46:56.302610 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.302522 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cd807e-82b9-4078-8f42-78b109ed0cf9" containerName="maas-api" Nov 04 01:46:56.302610 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.302530 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="e563c25f-bd6b-4a0d-a222-d874f9e8903f" containerName="payload-processing" Nov 04 01:46:56.305833 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.305813 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" Nov 04 01:46:56.356576 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.356545 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f4981c\"" Nov 04 01:46:56.404480 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.404434 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-d67ee732-dockercfg-k4778\"" Nov 04 01:46:56.406874 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.406840 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd"] Nov 04 01:46:56.442681 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.442643 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjvlx\" (UniqueName: \"kubernetes.io/projected/a1026273-2a93-4651-a078-3d159a294074-kube-api-access-sjvlx\") pod \"payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd\" (UID: \"a1026273-2a93-4651-a078-3d159a294074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" Nov 04 01:46:56.442859 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.442802 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1026273-2a93-4651-a078-3d159a294074-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd\" (UID: \"a1026273-2a93-4651-a078-3d159a294074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" Nov 04 01:46:56.453794 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.453767 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx"] Nov 04 01:46:56.457439 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.457423 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" Nov 04 01:46:56.485432 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.485403 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx"] Nov 04 01:46:56.543820 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.543785 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe0cf30-81cd-4ae9-8704-8424670f967d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx\" (UID: \"7fe0cf30-81cd-4ae9-8704-8424670f967d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" Nov 04 01:46:56.543997 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.543840 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvzsd\" (UniqueName: \"kubernetes.io/projected/7fe0cf30-81cd-4ae9-8704-8424670f967d-kube-api-access-mvzsd\") pod \"payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx\" (UID: \"7fe0cf30-81cd-4ae9-8704-8424670f967d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" Nov 04 01:46:56.543997 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.543963 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1026273-2a93-4651-a078-3d159a294074-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd\" (UID: \"a1026273-2a93-4651-a078-3d159a294074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" Nov 04 01:46:56.544076 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.544016 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvlx\" (UniqueName: \"kubernetes.io/projected/a1026273-2a93-4651-a078-3d159a294074-kube-api-access-sjvlx\") pod \"payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd\" (UID: \"a1026273-2a93-4651-a078-3d159a294074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" Nov 04 01:46:56.544811 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.544789 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1026273-2a93-4651-a078-3d159a294074-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd\" (UID: \"a1026273-2a93-4651-a078-3d159a294074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" Nov 04 01:46:56.600863 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.600828 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvlx\" (UniqueName: \"kubernetes.io/projected/a1026273-2a93-4651-a078-3d159a294074-kube-api-access-sjvlx\") pod \"payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd\" (UID: \"a1026273-2a93-4651-a078-3d159a294074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" Nov 04 01:46:56.619683 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.619653 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" Nov 04 01:46:56.644768 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.644728 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvzsd\" (UniqueName: \"kubernetes.io/projected/7fe0cf30-81cd-4ae9-8704-8424670f967d-kube-api-access-mvzsd\") pod \"payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx\" (UID: \"7fe0cf30-81cd-4ae9-8704-8424670f967d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" Nov 04 01:46:56.644953 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.644881 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe0cf30-81cd-4ae9-8704-8424670f967d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx\" (UID: \"7fe0cf30-81cd-4ae9-8704-8424670f967d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" Nov 04 01:46:56.645611 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.645593 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe0cf30-81cd-4ae9-8704-8424670f967d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx\" (UID: \"7fe0cf30-81cd-4ae9-8704-8424670f967d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" Nov 04 01:46:56.702065 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.702019 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvzsd\" (UniqueName: \"kubernetes.io/projected/7fe0cf30-81cd-4ae9-8704-8424670f967d-kube-api-access-mvzsd\") pod \"payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx\" (UID: \"7fe0cf30-81cd-4ae9-8704-8424670f967d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" Nov 04 01:46:56.770388 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.770352 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" Nov 04 01:46:56.986925 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:56.986840 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd"] Nov 04 01:46:56.994342 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:46:56.994311 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1026273_2a93_4651_a078_3d159a294074.slice/crio-1a94e10e0277ac5df31344a38b28c95265977bd2ca4a58563ad1818562fd7dc8 WatchSource:0}: Error finding container 1a94e10e0277ac5df31344a38b28c95265977bd2ca4a58563ad1818562fd7dc8: Status 404 returned error can't find the container with id 1a94e10e0277ac5df31344a38b28c95265977bd2ca4a58563ad1818562fd7dc8 Nov 04 01:46:57.142701 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:57.142663 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx"] Nov 04 01:46:57.148539 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:46:57.148512 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe0cf30_81cd_4ae9_8704_8424670f967d.slice/crio-fa64a279b1414493a38aaa3cdfb6e526e4b57f9bfc9abcf50938d282915c2e64 WatchSource:0}: Error finding container fa64a279b1414493a38aaa3cdfb6e526e4b57f9bfc9abcf50938d282915c2e64: Status 404 returned error can't find the container with id fa64a279b1414493a38aaa3cdfb6e526e4b57f9bfc9abcf50938d282915c2e64 Nov 04 01:46:57.855639 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:57.855602 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" event={"ID":"a1026273-2a93-4651-a078-3d159a294074","Type":"ContainerStarted","Data":"31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533"} Nov 04 01:46:57.856155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:57.855658 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" event={"ID":"a1026273-2a93-4651-a078-3d159a294074","Type":"ContainerStarted","Data":"1a94e10e0277ac5df31344a38b28c95265977bd2ca4a58563ad1818562fd7dc8"} Nov 04 01:46:57.856155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:57.855687 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" Nov 04 01:46:57.857024 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:57.856994 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" event={"ID":"7fe0cf30-81cd-4ae9-8704-8424670f967d","Type":"ContainerStarted","Data":"68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9"} Nov 04 01:46:57.857177 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:57.857040 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" event={"ID":"7fe0cf30-81cd-4ae9-8704-8424670f967d","Type":"ContainerStarted","Data":"fa64a279b1414493a38aaa3cdfb6e526e4b57f9bfc9abcf50938d282915c2e64"} Nov 04 01:46:57.857177 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:57.857072 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" Nov 04 01:46:57.967933 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:57.967873 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" podStartSLOduration=1.967858079 podStartE2EDuration="1.967858079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:46:57.966600247 +0000 UTC m=+2963.582946010" watchObservedRunningTime="2026-11-04 01:46:57.967858079 +0000 UTC m=+2963.584203841" Nov 04 01:46:58.098739 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.098667 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" podStartSLOduration=2.098644878 podStartE2EDuration="2.098644878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:46:58.092481792 +0000 UTC m=+2963.708827555" watchObservedRunningTime="2026-11-04 01:46:58.098644878 +0000 UTC m=+2963.714990654" Nov 04 01:46:58.318509 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.318427 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97"] Nov 04 01:46:58.322070 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.322047 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:58.379776 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.379730 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97"] Nov 04 01:46:58.383433 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.383402 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-f4981c-serving-cert\"" Nov 04 01:46:58.460567 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.460531 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8xsf\" (UniqueName: \"kubernetes.io/projected/c98059d1-7c15-4aa7-b33c-302f5461c6a6-kube-api-access-x8xsf\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:58.460762 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.460623 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98059d1-7c15-4aa7-b33c-302f5461c6a6-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:58.460762 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.460662 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:58.460762 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.460693 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:58.569186 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.565851 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8xsf\" (UniqueName: \"kubernetes.io/projected/c98059d1-7c15-4aa7-b33c-302f5461c6a6-kube-api-access-x8xsf\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:58.569186 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.565943 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98059d1-7c15-4aa7-b33c-302f5461c6a6-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:58.569186 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.565983 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:58.569186 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.566016 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:58.569186 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:46:58.566696 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-serving-cert: secret "maas-api-e2e-worker-single-main-f4981c-serving-cert" not found Nov 04 01:46:58.569186 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:46:58.566767 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-maas-api-tls podName:c98059d1-7c15-4aa7-b33c-302f5461c6a6 nodeName:}" failed. No retries permitted until 2026-11-04 01:46:59.066745004 +0000 UTC m=+2964.683090749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-maas-api-tls") pod "maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" (UID: "c98059d1-7c15-4aa7-b33c-302f5461c6a6") : secret "maas-api-e2e-worker-single-main-f4981c-serving-cert" not found Nov 04 01:46:58.569186 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.567879 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98059d1-7c15-4aa7-b33c-302f5461c6a6-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:58.575009 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.574737 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:58.623899 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:58.623859 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8xsf\" (UniqueName: \"kubernetes.io/projected/c98059d1-7c15-4aa7-b33c-302f5461c6a6-kube-api-access-x8xsf\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:59.071931 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:59.071891 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:59.074385 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:59.074353 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:59.235326 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:59.235291 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:46:59.604695 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:59.604662 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97"] Nov 04 01:46:59.611386 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:46:59.611355 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98059d1_7c15_4aa7_b33c_302f5461c6a6.slice/crio-b98534988748f66acd193ba604d691cbdc055df857e598c307aaee198fbb48d0 WatchSource:0}: Error finding container b98534988748f66acd193ba604d691cbdc055df857e598c307aaee198fbb48d0: Status 404 returned error can't find the container with id b98534988748f66acd193ba604d691cbdc055df857e598c307aaee198fbb48d0 Nov 04 01:46:59.868824 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:46:59.868740 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" event={"ID":"c98059d1-7c15-4aa7-b33c-302f5461c6a6","Type":"ContainerStarted","Data":"b98534988748f66acd193ba604d691cbdc055df857e598c307aaee198fbb48d0"} Nov 04 01:47:00.874587 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:47:00.874548 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" event={"ID":"c98059d1-7c15-4aa7-b33c-302f5461c6a6","Type":"ContainerStarted","Data":"4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38"} Nov 04 01:47:00.875055 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:47:00.874667 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:47:00.990918 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:47:00.990813 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" podStartSLOduration=1.948595911 podStartE2EDuration="2.990799019s" totalImagesPullingTime="1.042203108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:00.989565667 +0000 UTC m=+2966.605911430" watchObservedRunningTime="2026-11-04 01:47:00.990799019 +0000 UTC m=+2966.607144782" Nov 04 01:47:06.650173 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:47:06.650137 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:47:06.884223 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:47:06.884187 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:47:08.864593 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:47:08.864555 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" Nov 04 01:47:08.865033 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:47:08.864731 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" Nov 04 01:47:15.087766 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:47:15.087674 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:47:25.656597 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:47:25.656559 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:47:34.886768 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:47:34.886739 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:47:34.949327 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:47:34.949303 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:47:45.091336 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:47:45.091294 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:47:53.580556 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:47:53.580521 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:48:15.086733 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:48:15.086693 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:48:19.378332 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:19.378298 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:48:29.449854 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:29.449816 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx"] Nov 04 01:48:29.552239 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:29.552203 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd"] Nov 04 01:48:29.833286 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:29.833200 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97"] Nov 04 01:48:30.290225 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.290182 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" podUID="c98059d1-7c15-4aa7-b33c-302f5461c6a6" containerName="maas-api" containerID="cri-o://4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38" gracePeriod=30 Nov 04 01:48:30.548748 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.548690 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:48:30.614601 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.614571 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-maas-api-tls\") pod \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " Nov 04 01:48:30.614741 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.614617 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98059d1-7c15-4aa7-b33c-302f5461c6a6-kube-api-access-x8xsf\" (UniqueName: \"kubernetes.io/projected/c98059d1-7c15-4aa7-b33c-302f5461c6a6-kube-api-access-x8xsf\") pod \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " Nov 04 01:48:30.614812 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.614792 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98059d1-7c15-4aa7-b33c-302f5461c6a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98059d1-7c15-4aa7-b33c-302f5461c6a6-openshift-service-ca\") pod \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " Nov 04 01:48:30.614922 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.614838 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-metrics-certs\") pod \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\" (UID: \"c98059d1-7c15-4aa7-b33c-302f5461c6a6\") " Nov 04 01:48:30.615232 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.615206 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98059d1-7c15-4aa7-b33c-302f5461c6a6-openshift-service-ca" pod "c98059d1-7c15-4aa7-b33c-302f5461c6a6" (UID: "c98059d1-7c15-4aa7-b33c-302f5461c6a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:30.616556 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.616529 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-maas-api-tls" pod "c98059d1-7c15-4aa7-b33c-302f5461c6a6" (UID: "c98059d1-7c15-4aa7-b33c-302f5461c6a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:30.616746 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.616721 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98059d1-7c15-4aa7-b33c-302f5461c6a6-kube-api-access-x8xsf" pod "c98059d1-7c15-4aa7-b33c-302f5461c6a6" (UID: "c98059d1-7c15-4aa7-b33c-302f5461c6a6"). InnerVolumeSpecName "kube-api-access-x8xsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:30.616844 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.616751 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-metrics-certs" pod "c98059d1-7c15-4aa7-b33c-302f5461c6a6" (UID: "c98059d1-7c15-4aa7-b33c-302f5461c6a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:30.716059 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.715998 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:30.716059 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.716053 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8xsf\" (UniqueName: \"kubernetes.io/projected/c98059d1-7c15-4aa7-b33c-302f5461c6a6-kube-api-access-x8xsf\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:30.716059 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.716065 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c98059d1-7c15-4aa7-b33c-302f5461c6a6-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:30.716312 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:30.716074 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c98059d1-7c15-4aa7-b33c-302f5461c6a6-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:31.295825 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:31.295794 5786 generic.go:356] "Generic (PLEG): container finished" podID="c98059d1-7c15-4aa7-b33c-302f5461c6a6" containerID="4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38" exitCode=0 Nov 04 01:48:31.296014 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:31.295839 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" Nov 04 01:48:31.296014 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:31.295861 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" event={"ID":"c98059d1-7c15-4aa7-b33c-302f5461c6a6","Type":"ContainerDied","Data":"4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38"} Nov 04 01:48:31.296014 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:31.295908 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97" event={"ID":"c98059d1-7c15-4aa7-b33c-302f5461c6a6","Type":"ContainerDied","Data":"b98534988748f66acd193ba604d691cbdc055df857e598c307aaee198fbb48d0"} Nov 04 01:48:31.296014 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:31.295931 5786 scope.go:122] "RemoveContainer" containerID="4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38" Nov 04 01:48:31.305994 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:31.305973 5786 scope.go:122] "RemoveContainer" containerID="4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38" Nov 04 01:48:31.306300 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:48:31.306279 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38\": container with ID starting with 4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38 not found: ID does not exist" containerID="4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38" Nov 04 01:48:31.306384 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:31.306312 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38"} err="failed to get container status \"4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38\": rpc error: code = NotFound desc = could not find container \"4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38\": container with ID starting with 4ae7c7bf4dbe0dfc8449e9bea4e729fca046fec9984e7258fd3f7a6f96ed3f38 not found: ID does not exist" Nov 04 01:48:31.453817 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:31.453784 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97"] Nov 04 01:48:31.501511 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:31.501477 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f4981c-78bbd65985-74c97"] Nov 04 01:48:32.504754 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:32.504723 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98059d1-7c15-4aa7-b33c-302f5461c6a6" path="/var/lib/kubelet/pods/c98059d1-7c15-4aa7-b33c-302f5461c6a6/volumes" Nov 04 01:48:34.451189 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.451100 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" podUID="7fe0cf30-81cd-4ae9-8704-8424670f967d" containerName="payload-processing" containerID="cri-o://68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9" gracePeriod=25 Nov 04 01:48:34.552907 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.552854 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" podUID="a1026273-2a93-4651-a078-3d159a294074" containerName="payload-pre-processing" containerID="cri-o://31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533" gracePeriod=25 Nov 04 01:48:34.752904 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.752876 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" Nov 04 01:48:34.866103 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.866073 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:48:34.870366 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.870346 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" Nov 04 01:48:34.877355 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.877329 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe0cf30-81cd-4ae9-8704-8424670f967d-kube-api-access-mvzsd\" (UniqueName: \"kubernetes.io/projected/7fe0cf30-81cd-4ae9-8704-8424670f967d-kube-api-access-mvzsd\") pod \"7fe0cf30-81cd-4ae9-8704-8424670f967d\" (UID: \"7fe0cf30-81cd-4ae9-8704-8424670f967d\") " Nov 04 01:48:34.877473 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.877429 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe0cf30-81cd-4ae9-8704-8424670f967d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe0cf30-81cd-4ae9-8704-8424670f967d-plugins-config-volume\") pod \"7fe0cf30-81cd-4ae9-8704-8424670f967d\" (UID: \"7fe0cf30-81cd-4ae9-8704-8424670f967d\") " Nov 04 01:48:34.877859 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.877838 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe0cf30-81cd-4ae9-8704-8424670f967d-plugins-config-volume" pod "7fe0cf30-81cd-4ae9-8704-8424670f967d" (UID: "7fe0cf30-81cd-4ae9-8704-8424670f967d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:34.879776 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.879736 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe0cf30-81cd-4ae9-8704-8424670f967d-kube-api-access-mvzsd" pod "7fe0cf30-81cd-4ae9-8704-8424670f967d" (UID: "7fe0cf30-81cd-4ae9-8704-8424670f967d"). InnerVolumeSpecName "kube-api-access-mvzsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:34.978537 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.978466 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1026273-2a93-4651-a078-3d159a294074-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1026273-2a93-4651-a078-3d159a294074-plugins-config-volume\") pod \"a1026273-2a93-4651-a078-3d159a294074\" (UID: \"a1026273-2a93-4651-a078-3d159a294074\") " Nov 04 01:48:34.978681 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.978540 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1026273-2a93-4651-a078-3d159a294074-kube-api-access-sjvlx\" (UniqueName: \"kubernetes.io/projected/a1026273-2a93-4651-a078-3d159a294074-kube-api-access-sjvlx\") pod \"a1026273-2a93-4651-a078-3d159a294074\" (UID: \"a1026273-2a93-4651-a078-3d159a294074\") " Nov 04 01:48:34.978773 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.978760 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvzsd\" (UniqueName: \"kubernetes.io/projected/7fe0cf30-81cd-4ae9-8704-8424670f967d-kube-api-access-mvzsd\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:34.978837 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.978774 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe0cf30-81cd-4ae9-8704-8424670f967d-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:34.978902 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.978836 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1026273-2a93-4651-a078-3d159a294074-plugins-config-volume" pod "a1026273-2a93-4651-a078-3d159a294074" (UID: "a1026273-2a93-4651-a078-3d159a294074"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:34.980976 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:34.980952 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1026273-2a93-4651-a078-3d159a294074-kube-api-access-sjvlx" pod "a1026273-2a93-4651-a078-3d159a294074" (UID: "a1026273-2a93-4651-a078-3d159a294074"). InnerVolumeSpecName "kube-api-access-sjvlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:35.079849 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.079812 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjvlx\" (UniqueName: \"kubernetes.io/projected/a1026273-2a93-4651-a078-3d159a294074-kube-api-access-sjvlx\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:35.079849 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.079841 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1026273-2a93-4651-a078-3d159a294074-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:35.315742 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.315655 5786 generic.go:356] "Generic (PLEG): container finished" podID="a1026273-2a93-4651-a078-3d159a294074" containerID="31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533" exitCode=0 Nov 04 01:48:35.315742 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.315704 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" Nov 04 01:48:35.315742 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.315733 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" event={"ID":"a1026273-2a93-4651-a078-3d159a294074","Type":"ContainerDied","Data":"31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533"} Nov 04 01:48:35.316017 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.315783 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd" event={"ID":"a1026273-2a93-4651-a078-3d159a294074","Type":"ContainerDied","Data":"1a94e10e0277ac5df31344a38b28c95265977bd2ca4a58563ad1818562fd7dc8"} Nov 04 01:48:35.316017 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.315811 5786 scope.go:122] "RemoveContainer" containerID="31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533" Nov 04 01:48:35.317111 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.317087 5786 generic.go:356] "Generic (PLEG): container finished" podID="7fe0cf30-81cd-4ae9-8704-8424670f967d" containerID="68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9" exitCode=0 Nov 04 01:48:35.317241 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.317149 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" Nov 04 01:48:35.317241 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.317147 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" event={"ID":"7fe0cf30-81cd-4ae9-8704-8424670f967d","Type":"ContainerDied","Data":"68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9"} Nov 04 01:48:35.317241 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.317231 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx" event={"ID":"7fe0cf30-81cd-4ae9-8704-8424670f967d","Type":"ContainerDied","Data":"fa64a279b1414493a38aaa3cdfb6e526e4b57f9bfc9abcf50938d282915c2e64"} Nov 04 01:48:35.325895 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.325879 5786 scope.go:122] "RemoveContainer" containerID="31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533" Nov 04 01:48:35.326198 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:48:35.326178 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533\": container with ID starting with 31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533 not found: ID does not exist" containerID="31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533" Nov 04 01:48:35.326292 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.326208 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533"} err="failed to get container status \"31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533\": rpc error: code = NotFound desc = could not find container \"31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533\": container with ID starting with 31808d6f28d92003ac9c4d96f8e54e7a5cb05320659cadca44a85574e21ff533 not found: ID does not exist" Nov 04 01:48:35.326292 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.326227 5786 scope.go:122] "RemoveContainer" containerID="68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9" Nov 04 01:48:35.334100 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.334082 5786 scope.go:122] "RemoveContainer" containerID="68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9" Nov 04 01:48:35.334383 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:48:35.334364 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9\": container with ID starting with 68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9 not found: ID does not exist" containerID="68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9" Nov 04 01:48:35.334439 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.334389 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9"} err="failed to get container status \"68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9\": rpc error: code = NotFound desc = could not find container \"68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9\": container with ID starting with 68f8556dd6b64804a1df8484ffa4d40ba08853181f392c7e7d61e563dca163c9 not found: ID does not exist" Nov 04 01:48:35.483386 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.483365 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd"] Nov 04 01:48:35.489689 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.489669 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f4981c-c47f5j8wwd"] Nov 04 01:48:35.594938 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.594912 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx"] Nov 04 01:48:35.602040 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:35.602010 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f4981c-54cdfc4cf4tvhx"] Nov 04 01:48:36.504591 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:36.504563 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe0cf30-81cd-4ae9-8704-8424670f967d" path="/var/lib/kubelet/pods/7fe0cf30-81cd-4ae9-8704-8424670f967d/volumes" Nov 04 01:48:36.504962 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:36.504909 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1026273-2a93-4651-a078-3d159a294074" path="/var/lib/kubelet/pods/a1026273-2a93-4651-a078-3d159a294074/volumes" Nov 04 01:48:45.087030 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:48:45.086951 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:48:53.561345 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:48:53.561311 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:49:15.086735 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:49:15.086697 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:49:38.263048 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:49:38.263011 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:49:45.087504 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:49:45.087463 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:49:56.737758 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:49:56.737721 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:50:12.550002 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:50:12.549966 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:50:15.183449 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:50:15.183355 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:50:27.993051 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:50:27.993007 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:50:40.340862 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:50:40.340823 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:50:45.086671 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:50:45.086632 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:50:51.767488 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:50:51.767451 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:51:13.062001 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.061953 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:51:13.374694 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.374660 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5"] Nov 04 01:51:13.375070 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.375057 5786 cpu_manager.go:414] "containerMap: removing container" podUID="7fe0cf30-81cd-4ae9-8704-8424670f967d" containerName="payload-processing" Nov 04 01:51:13.375070 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.375070 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe0cf30-81cd-4ae9-8704-8424670f967d" containerName="payload-processing" Nov 04 01:51:13.375269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.375084 5786 cpu_manager.go:414] "containerMap: removing container" podUID="a1026273-2a93-4651-a078-3d159a294074" containerName="payload-pre-processing" Nov 04 01:51:13.375269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.375090 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1026273-2a93-4651-a078-3d159a294074" containerName="payload-pre-processing" Nov 04 01:51:13.375269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.375108 5786 cpu_manager.go:414] "containerMap: removing container" podUID="c98059d1-7c15-4aa7-b33c-302f5461c6a6" containerName="maas-api" Nov 04 01:51:13.375269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.375113 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98059d1-7c15-4aa7-b33c-302f5461c6a6" containerName="maas-api" Nov 04 01:51:13.375269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.375184 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe0cf30-81cd-4ae9-8704-8424670f967d" containerName="payload-processing" Nov 04 01:51:13.375269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.375193 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1026273-2a93-4651-a078-3d159a294074" containerName="payload-pre-processing" Nov 04 01:51:13.375269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.375201 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98059d1-7c15-4aa7-b33c-302f5461c6a6" containerName="maas-api" Nov 04 01:51:13.378232 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.378212 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.500155 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.489964 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5"] Nov 04 01:51:13.509999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.509961 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a4db80-e943-4e66-9990-2d07ecdcbfa7-maas-api-tls\") pod \"maas-api-6fc6d86564-2p9h5\" (UID: \"63a4db80-e943-4e66-9990-2d07ecdcbfa7\") " pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.510326 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.510307 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a4db80-e943-4e66-9990-2d07ecdcbfa7-openshift-service-ca\") pod \"maas-api-6fc6d86564-2p9h5\" (UID: \"63a4db80-e943-4e66-9990-2d07ecdcbfa7\") " pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.510530 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.510516 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsbvf\" (UniqueName: \"kubernetes.io/projected/63a4db80-e943-4e66-9990-2d07ecdcbfa7-kube-api-access-nsbvf\") pod \"maas-api-6fc6d86564-2p9h5\" (UID: \"63a4db80-e943-4e66-9990-2d07ecdcbfa7\") " pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.510710 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.510697 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63a4db80-e943-4e66-9990-2d07ecdcbfa7-metrics-certs\") pod \"maas-api-6fc6d86564-2p9h5\" (UID: \"63a4db80-e943-4e66-9990-2d07ecdcbfa7\") " pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.611666 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.611627 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63a4db80-e943-4e66-9990-2d07ecdcbfa7-metrics-certs\") pod \"maas-api-6fc6d86564-2p9h5\" (UID: \"63a4db80-e943-4e66-9990-2d07ecdcbfa7\") " pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.611869 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.611699 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a4db80-e943-4e66-9990-2d07ecdcbfa7-maas-api-tls\") pod \"maas-api-6fc6d86564-2p9h5\" (UID: \"63a4db80-e943-4e66-9990-2d07ecdcbfa7\") " pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.611869 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.611731 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a4db80-e943-4e66-9990-2d07ecdcbfa7-openshift-service-ca\") pod \"maas-api-6fc6d86564-2p9h5\" (UID: \"63a4db80-e943-4e66-9990-2d07ecdcbfa7\") " pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.611869 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.611760 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsbvf\" (UniqueName: \"kubernetes.io/projected/63a4db80-e943-4e66-9990-2d07ecdcbfa7-kube-api-access-nsbvf\") pod \"maas-api-6fc6d86564-2p9h5\" (UID: \"63a4db80-e943-4e66-9990-2d07ecdcbfa7\") " pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.612478 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.612445 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a4db80-e943-4e66-9990-2d07ecdcbfa7-openshift-service-ca\") pod \"maas-api-6fc6d86564-2p9h5\" (UID: \"63a4db80-e943-4e66-9990-2d07ecdcbfa7\") " pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.614104 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.614077 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63a4db80-e943-4e66-9990-2d07ecdcbfa7-metrics-certs\") pod \"maas-api-6fc6d86564-2p9h5\" (UID: \"63a4db80-e943-4e66-9990-2d07ecdcbfa7\") " pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.614214 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.614141 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a4db80-e943-4e66-9990-2d07ecdcbfa7-maas-api-tls\") pod \"maas-api-6fc6d86564-2p9h5\" (UID: \"63a4db80-e943-4e66-9990-2d07ecdcbfa7\") " pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.666451 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.666378 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsbvf\" (UniqueName: \"kubernetes.io/projected/63a4db80-e943-4e66-9990-2d07ecdcbfa7-kube-api-access-nsbvf\") pod \"maas-api-6fc6d86564-2p9h5\" (UID: \"63a4db80-e943-4e66-9990-2d07ecdcbfa7\") " pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:13.691325 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:13.691294 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:14.060101 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:14.060022 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5"] Nov 04 01:51:14.065942 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:51:14.065914 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a4db80_e943_4e66_9990_2d07ecdcbfa7.slice/crio-853bca25feca94460fa47066a998ca9eda8daf7f930254f1172092213ff6689b WatchSource:0}: Error finding container 853bca25feca94460fa47066a998ca9eda8daf7f930254f1172092213ff6689b: Status 404 returned error can't find the container with id 853bca25feca94460fa47066a998ca9eda8daf7f930254f1172092213ff6689b Nov 04 01:51:14.067337 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:14.067319 5786 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:51:14.179946 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:14.179908 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" event={"ID":"63a4db80-e943-4e66-9990-2d07ecdcbfa7","Type":"ContainerStarted","Data":"853bca25feca94460fa47066a998ca9eda8daf7f930254f1172092213ff6689b"} Nov 04 01:51:15.089095 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:51:15.089062 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:51:15.186012 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:15.185977 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" event={"ID":"63a4db80-e943-4e66-9990-2d07ecdcbfa7","Type":"ContainerStarted","Data":"2986397afd79ca32b26a529765147b6c25784cdaff0ef81638fef0a85da9fbe7"} Nov 04 01:51:15.186245 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:15.186092 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:15.304767 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:15.304666 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" podStartSLOduration=1.335121817 podStartE2EDuration="2.3046522s" totalImagesPullingTime="969.530383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:51:15.303436209 +0000 UTC m=+3220.919781972" watchObservedRunningTime="2026-11-04 01:51:15.3046522 +0000 UTC m=+3220.920997960" Nov 04 01:51:21.196234 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:21.196203 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6fc6d86564-2p9h5" Nov 04 01:51:27.639251 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.639210 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf"] Nov 04 01:51:27.645143 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.644915 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" Nov 04 01:51:27.650474 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.650440 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz"] Nov 04 01:51:27.654580 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.654549 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:27.665900 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.665871 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxmx2\" (UniqueName: \"kubernetes.io/projected/25946ee4-d876-4e68-ba76-319c840636fc-kube-api-access-vxmx2\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:27.666029 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.665918 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-maas-api-tls\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:27.666029 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.665986 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-metrics-certs\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:27.666109 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.666080 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9dgb\" (UniqueName: \"kubernetes.io/projected/0cada49e-dbf8-48b4-8e66-9bd5e9512721-kube-api-access-b9dgb\") pod \"payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf\" (UID: \"0cada49e-dbf8-48b4-8e66-9bd5e9512721\") " pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" Nov 04 01:51:27.666174 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.666158 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cada49e-dbf8-48b4-8e66-9bd5e9512721-plugins-config-volume\") pod \"payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf\" (UID: \"0cada49e-dbf8-48b4-8e66-9bd5e9512721\") " pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" Nov 04 01:51:27.666234 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.666210 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25946ee4-d876-4e68-ba76-319c840636fc-openshift-service-ca\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:27.696538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.696109 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4955ec16\"" Nov 04 01:51:27.696538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.696437 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4955ec16-dockercfg-gqxz2\"" Nov 04 01:51:27.696538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.696445 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf"] Nov 04 01:51:27.742329 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.742299 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4955ec16-serving-cert\"" Nov 04 01:51:27.744661 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.744635 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz"] Nov 04 01:51:27.767374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.767342 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9dgb\" (UniqueName: \"kubernetes.io/projected/0cada49e-dbf8-48b4-8e66-9bd5e9512721-kube-api-access-b9dgb\") pod \"payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf\" (UID: \"0cada49e-dbf8-48b4-8e66-9bd5e9512721\") " pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" Nov 04 01:51:27.767542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.767395 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cada49e-dbf8-48b4-8e66-9bd5e9512721-plugins-config-volume\") pod \"payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf\" (UID: \"0cada49e-dbf8-48b4-8e66-9bd5e9512721\") " pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" Nov 04 01:51:27.767542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.767423 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25946ee4-d876-4e68-ba76-319c840636fc-openshift-service-ca\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:27.767542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.767463 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxmx2\" (UniqueName: \"kubernetes.io/projected/25946ee4-d876-4e68-ba76-319c840636fc-kube-api-access-vxmx2\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:27.767542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.767492 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-maas-api-tls\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:27.767542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.767521 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-metrics-certs\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:27.767818 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:51:27.767642 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-serving-cert: secret "maas-api-e2e-ait-4955ec16-serving-cert" not found Nov 04 01:51:27.767818 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:51:27.767736 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-maas-api-tls podName:25946ee4-d876-4e68-ba76-319c840636fc nodeName:}" failed. No retries permitted until 2026-11-04 01:51:28.267711558 +0000 UTC m=+3233.884057316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-maas-api-tls") pod "maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" (UID: "25946ee4-d876-4e68-ba76-319c840636fc") : secret "maas-api-e2e-ait-4955ec16-serving-cert" not found Nov 04 01:51:27.768172 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.768151 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25946ee4-d876-4e68-ba76-319c840636fc-openshift-service-ca\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:27.768314 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.768297 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cada49e-dbf8-48b4-8e66-9bd5e9512721-plugins-config-volume\") pod \"payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf\" (UID: \"0cada49e-dbf8-48b4-8e66-9bd5e9512721\") " pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" Nov 04 01:51:27.769942 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.769925 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-metrics-certs\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:27.793714 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.793675 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54"] Nov 04 01:51:27.798508 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.798468 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" Nov 04 01:51:27.824716 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.824673 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxmx2\" (UniqueName: \"kubernetes.io/projected/25946ee4-d876-4e68-ba76-319c840636fc-kube-api-access-vxmx2\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:27.824716 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.824704 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9dgb\" (UniqueName: \"kubernetes.io/projected/0cada49e-dbf8-48b4-8e66-9bd5e9512721-kube-api-access-b9dgb\") pod \"payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf\" (UID: \"0cada49e-dbf8-48b4-8e66-9bd5e9512721\") " pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" Nov 04 01:51:27.851184 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.851151 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54"] Nov 04 01:51:27.868948 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.868906 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zpjg\" (UniqueName: \"kubernetes.io/projected/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-kube-api-access-8zpjg\") pod \"payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54\" (UID: \"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" Nov 04 01:51:27.869166 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.869066 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54\" (UID: \"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" Nov 04 01:51:27.960780 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.960691 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" Nov 04 01:51:27.970158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.969829 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zpjg\" (UniqueName: \"kubernetes.io/projected/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-kube-api-access-8zpjg\") pod \"payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54\" (UID: \"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" Nov 04 01:51:27.970351 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.970325 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54\" (UID: \"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" Nov 04 01:51:27.971032 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:27.971007 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54\" (UID: \"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" Nov 04 01:51:28.025679 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:28.025654 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zpjg\" (UniqueName: \"kubernetes.io/projected/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-kube-api-access-8zpjg\") pod \"payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54\" (UID: \"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" Nov 04 01:51:28.113408 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:28.113367 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" Nov 04 01:51:28.275913 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:28.275816 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-maas-api-tls\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:28.278291 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:28.278270 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-maas-api-tls\") pod \"maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:28.333789 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:28.333752 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf"] Nov 04 01:51:28.339175 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:51:28.339151 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cada49e_dbf8_48b4_8e66_9bd5e9512721.slice/crio-c27420bf3d5e4e176a4c5dd4d8cecf87b3e9910fb68563a19cf952f3a2638b84 WatchSource:0}: Error finding container c27420bf3d5e4e176a4c5dd4d8cecf87b3e9910fb68563a19cf952f3a2638b84: Status 404 returned error can't find the container with id c27420bf3d5e4e176a4c5dd4d8cecf87b3e9910fb68563a19cf952f3a2638b84 Nov 04 01:51:28.481623 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:28.481582 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54"] Nov 04 01:51:28.487511 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:51:28.487479 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae97c0b_66e3_43cf_b6ef_03bb6e80455f.slice/crio-7f370548780a636ac94916b5bc691fe2267bd144089d092bac836e1ee49baa13 WatchSource:0}: Error finding container 7f370548780a636ac94916b5bc691fe2267bd144089d092bac836e1ee49baa13: Status 404 returned error can't find the container with id 7f370548780a636ac94916b5bc691fe2267bd144089d092bac836e1ee49baa13 Nov 04 01:51:28.570777 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:28.570747 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:28.945927 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:28.945887 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz"] Nov 04 01:51:28.951268 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:51:28.951239 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25946ee4_d876_4e68_ba76_319c840636fc.slice/crio-591300220beb59ac24a177c5a6652af576cbcbdf8e4ce30c6dc86d445a067fbd WatchSource:0}: Error finding container 591300220beb59ac24a177c5a6652af576cbcbdf8e4ce30c6dc86d445a067fbd: Status 404 returned error can't find the container with id 591300220beb59ac24a177c5a6652af576cbcbdf8e4ce30c6dc86d445a067fbd Nov 04 01:51:29.251826 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:29.251735 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" event={"ID":"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f","Type":"ContainerStarted","Data":"faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11"} Nov 04 01:51:29.251988 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:29.251893 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" Nov 04 01:51:29.251988 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:29.251938 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" event={"ID":"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f","Type":"ContainerStarted","Data":"7f370548780a636ac94916b5bc691fe2267bd144089d092bac836e1ee49baa13"} Nov 04 01:51:29.253078 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:29.253048 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" event={"ID":"25946ee4-d876-4e68-ba76-319c840636fc","Type":"ContainerStarted","Data":"591300220beb59ac24a177c5a6652af576cbcbdf8e4ce30c6dc86d445a067fbd"} Nov 04 01:51:29.254441 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:29.254419 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" event={"ID":"0cada49e-dbf8-48b4-8e66-9bd5e9512721","Type":"ContainerStarted","Data":"26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd"} Nov 04 01:51:29.254542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:29.254456 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" event={"ID":"0cada49e-dbf8-48b4-8e66-9bd5e9512721","Type":"ContainerStarted","Data":"c27420bf3d5e4e176a4c5dd4d8cecf87b3e9910fb68563a19cf952f3a2638b84"} Nov 04 01:51:29.254542 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:29.254519 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" Nov 04 01:51:29.363638 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:29.363585 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" podStartSLOduration=2.363572594 podStartE2EDuration="2.363572594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:51:29.361895081 +0000 UTC m=+3234.978240843" watchObservedRunningTime="2026-11-04 01:51:29.363572594 +0000 UTC m=+3234.979918357" Nov 04 01:51:29.472750 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:29.472685 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" podStartSLOduration=2.472669767 podStartE2EDuration="2.472669767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:51:29.47189413 +0000 UTC m=+3235.088239894" watchObservedRunningTime="2026-11-04 01:51:29.472669767 +0000 UTC m=+3235.089015556" Nov 04 01:51:29.538807 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:29.538720 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2"] Nov 04 01:51:29.538985 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:29.538961 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" podUID="d85d60f5-b598-4909-bffd-b383252a2be5" containerName="maas-api" containerID="cri-o://ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88" gracePeriod=30 Nov 04 01:51:29.599287 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:29.599245 5786 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" podUID="d85d60f5-b598-4909-bffd-b383252a2be5" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.32:8443/health\": dial tcp 10.132.0.32:8443: connect: connection refused" Nov 04 01:51:30.260997 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:30.260896 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" event={"ID":"25946ee4-d876-4e68-ba76-319c840636fc","Type":"ContainerStarted","Data":"adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7"} Nov 04 01:51:30.261465 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:30.261357 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:30.375835 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:30.375778 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" podStartSLOduration=2.43801818 podStartE2EDuration="3.375763742s" totalImagesPullingTime="937.745562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:51:30.374855193 +0000 UTC m=+3235.991200957" watchObservedRunningTime="2026-11-04 01:51:30.375763742 +0000 UTC m=+3235.992109504" Nov 04 01:51:30.813911 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:30.813882 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:51:30.917462 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:30.917427 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85d60f5-b598-4909-bffd-b383252a2be5-kube-api-access-zc2k5\" (UniqueName: \"kubernetes.io/projected/d85d60f5-b598-4909-bffd-b383252a2be5-kube-api-access-zc2k5\") pod \"d85d60f5-b598-4909-bffd-b383252a2be5\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " Nov 04 01:51:30.917644 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:30.917470 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-maas-api-tls\") pod \"d85d60f5-b598-4909-bffd-b383252a2be5\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " Nov 04 01:51:30.917644 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:30.917537 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85d60f5-b598-4909-bffd-b383252a2be5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85d60f5-b598-4909-bffd-b383252a2be5-openshift-service-ca\") pod \"d85d60f5-b598-4909-bffd-b383252a2be5\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " Nov 04 01:51:30.917644 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:30.917618 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-metrics-certs\") pod \"d85d60f5-b598-4909-bffd-b383252a2be5\" (UID: \"d85d60f5-b598-4909-bffd-b383252a2be5\") " Nov 04 01:51:30.917976 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:30.917942 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85d60f5-b598-4909-bffd-b383252a2be5-openshift-service-ca" pod "d85d60f5-b598-4909-bffd-b383252a2be5" (UID: "d85d60f5-b598-4909-bffd-b383252a2be5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:51:30.919718 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:30.919690 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85d60f5-b598-4909-bffd-b383252a2be5-kube-api-access-zc2k5" pod "d85d60f5-b598-4909-bffd-b383252a2be5" (UID: "d85d60f5-b598-4909-bffd-b383252a2be5"). InnerVolumeSpecName "kube-api-access-zc2k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:51:30.919829 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:30.919696 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-maas-api-tls" pod "d85d60f5-b598-4909-bffd-b383252a2be5" (UID: "d85d60f5-b598-4909-bffd-b383252a2be5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:51:30.919829 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:30.919760 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-metrics-certs" pod "d85d60f5-b598-4909-bffd-b383252a2be5" (UID: "d85d60f5-b598-4909-bffd-b383252a2be5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:51:31.018964 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.018924 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85d60f5-b598-4909-bffd-b383252a2be5-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:31.018964 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.018956 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:31.018964 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.018966 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc2k5\" (UniqueName: \"kubernetes.io/projected/d85d60f5-b598-4909-bffd-b383252a2be5-kube-api-access-zc2k5\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:31.018964 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.018975 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d85d60f5-b598-4909-bffd-b383252a2be5-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:31.267572 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.267474 5786 generic.go:356] "Generic (PLEG): container finished" podID="d85d60f5-b598-4909-bffd-b383252a2be5" containerID="ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88" exitCode=0 Nov 04 01:51:31.267572 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.267538 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" Nov 04 01:51:31.268064 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.267553 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" event={"ID":"d85d60f5-b598-4909-bffd-b383252a2be5","Type":"ContainerDied","Data":"ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88"} Nov 04 01:51:31.268064 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.267646 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2" event={"ID":"d85d60f5-b598-4909-bffd-b383252a2be5","Type":"ContainerDied","Data":"8f94f7e40973d534012d60db7bd0f4c9e961f1b679f0382617f0fe75fded8327"} Nov 04 01:51:31.268064 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.267668 5786 scope.go:122] "RemoveContainer" containerID="ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88" Nov 04 01:51:31.277317 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.277298 5786 scope.go:122] "RemoveContainer" containerID="ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88" Nov 04 01:51:31.277612 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:51:31.277592 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88\": container with ID starting with ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88 not found: ID does not exist" containerID="ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88" Nov 04 01:51:31.277653 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.277621 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88"} err="failed to get container status \"ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88\": rpc error: code = NotFound desc = could not find container \"ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88\": container with ID starting with ccbd1d23b8f33b38f62ba58ae6a8c874cd2bc8136f07e3ab0a86d23a0291bc88 not found: ID does not exist" Nov 04 01:51:31.424728 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.424696 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2"] Nov 04 01:51:31.428168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:31.428139 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7894895fc6-q8jq2"] Nov 04 01:51:32.505880 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:32.505848 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85d60f5-b598-4909-bffd-b383252a2be5" path="/var/lib/kubelet/pods/d85d60f5-b598-4909-bffd-b383252a2be5/volumes" Nov 04 01:51:35.935416 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:35.935380 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:51:36.273794 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:36.273719 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:51:40.264356 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:40.264326 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" Nov 04 01:51:40.264841 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:51:40.264397 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" Nov 04 01:51:45.086656 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:51:45.086576 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:52:06.689073 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:06.689037 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf"] Nov 04 01:52:06.796819 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:06.796780 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54"] Nov 04 01:52:11.690010 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:11.689962 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" podUID="0cada49e-dbf8-48b4-8e66-9bd5e9512721" containerName="payload-processing" containerID="cri-o://26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd" gracePeriod=25 Nov 04 01:52:11.798307 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:11.798248 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" podUID="3ae97c0b-66e3-43cf-b6ef-03bb6e80455f" containerName="payload-pre-processing" containerID="cri-o://faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11" gracePeriod=25 Nov 04 01:52:11.947191 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:11.947108 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" Nov 04 01:52:12.024847 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.024816 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cada49e-dbf8-48b4-8e66-9bd5e9512721-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cada49e-dbf8-48b4-8e66-9bd5e9512721-plugins-config-volume\") pod \"0cada49e-dbf8-48b4-8e66-9bd5e9512721\" (UID: \"0cada49e-dbf8-48b4-8e66-9bd5e9512721\") " Nov 04 01:52:12.025039 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.024862 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cada49e-dbf8-48b4-8e66-9bd5e9512721-kube-api-access-b9dgb\" (UniqueName: \"kubernetes.io/projected/0cada49e-dbf8-48b4-8e66-9bd5e9512721-kube-api-access-b9dgb\") pod \"0cada49e-dbf8-48b4-8e66-9bd5e9512721\" (UID: \"0cada49e-dbf8-48b4-8e66-9bd5e9512721\") " Nov 04 01:52:12.025284 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.025261 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cada49e-dbf8-48b4-8e66-9bd5e9512721-plugins-config-volume" pod "0cada49e-dbf8-48b4-8e66-9bd5e9512721" (UID: "0cada49e-dbf8-48b4-8e66-9bd5e9512721"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:52:12.027079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.027046 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cada49e-dbf8-48b4-8e66-9bd5e9512721-kube-api-access-b9dgb" pod "0cada49e-dbf8-48b4-8e66-9bd5e9512721" (UID: "0cada49e-dbf8-48b4-8e66-9bd5e9512721"). InnerVolumeSpecName "kube-api-access-b9dgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:52:12.062486 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.062460 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" Nov 04 01:52:12.125549 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.125517 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-kube-api-access-8zpjg\" (UniqueName: \"kubernetes.io/projected/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-kube-api-access-8zpjg\") pod \"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f\" (UID: \"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f\") " Nov 04 01:52:12.125762 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.125603 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-plugins-config-volume\") pod \"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f\" (UID: \"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f\") " Nov 04 01:52:12.125902 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.125883 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cada49e-dbf8-48b4-8e66-9bd5e9512721-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:12.125964 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.125906 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9dgb\" (UniqueName: \"kubernetes.io/projected/0cada49e-dbf8-48b4-8e66-9bd5e9512721-kube-api-access-b9dgb\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:12.125964 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.125939 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-plugins-config-volume" pod "3ae97c0b-66e3-43cf-b6ef-03bb6e80455f" (UID: "3ae97c0b-66e3-43cf-b6ef-03bb6e80455f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:52:12.127634 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.127610 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-kube-api-access-8zpjg" pod "3ae97c0b-66e3-43cf-b6ef-03bb6e80455f" (UID: "3ae97c0b-66e3-43cf-b6ef-03bb6e80455f"). InnerVolumeSpecName "kube-api-access-8zpjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:52:12.226612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.226531 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:12.226612 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.226560 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zpjg\" (UniqueName: \"kubernetes.io/projected/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f-kube-api-access-8zpjg\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:12.467188 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.467148 5786 generic.go:356] "Generic (PLEG): container finished" podID="0cada49e-dbf8-48b4-8e66-9bd5e9512721" containerID="26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd" exitCode=0 Nov 04 01:52:12.467373 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.467193 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" Nov 04 01:52:12.467373 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.467225 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" event={"ID":"0cada49e-dbf8-48b4-8e66-9bd5e9512721","Type":"ContainerDied","Data":"26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd"} Nov 04 01:52:12.467373 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.467282 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf" event={"ID":"0cada49e-dbf8-48b4-8e66-9bd5e9512721","Type":"ContainerDied","Data":"c27420bf3d5e4e176a4c5dd4d8cecf87b3e9910fb68563a19cf952f3a2638b84"} Nov 04 01:52:12.467373 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.467307 5786 scope.go:122] "RemoveContainer" containerID="26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd" Nov 04 01:52:12.468737 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.468710 5786 generic.go:356] "Generic (PLEG): container finished" podID="3ae97c0b-66e3-43cf-b6ef-03bb6e80455f" containerID="faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11" exitCode=0 Nov 04 01:52:12.468870 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.468738 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" Nov 04 01:52:12.468870 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.468757 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" event={"ID":"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f","Type":"ContainerDied","Data":"faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11"} Nov 04 01:52:12.468870 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.468795 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54" event={"ID":"3ae97c0b-66e3-43cf-b6ef-03bb6e80455f","Type":"ContainerDied","Data":"7f370548780a636ac94916b5bc691fe2267bd144089d092bac836e1ee49baa13"} Nov 04 01:52:12.477756 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.477621 5786 scope.go:122] "RemoveContainer" containerID="26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd" Nov 04 01:52:12.478010 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:52:12.477981 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd\": container with ID starting with 26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd not found: ID does not exist" containerID="26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd" Nov 04 01:52:12.478079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.478022 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd"} err="failed to get container status \"26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd\": rpc error: code = NotFound desc = could not find container \"26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd\": container with ID starting with 26a000026522c4219ef7db8e274be0ad0d9a8efe8b5c3a023abe153d1e8a17cd not found: ID does not exist" Nov 04 01:52:12.478079 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.478045 5786 scope.go:122] "RemoveContainer" containerID="faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11" Nov 04 01:52:12.486128 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.486098 5786 scope.go:122] "RemoveContainer" containerID="faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11" Nov 04 01:52:12.486477 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:52:12.486454 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11\": container with ID starting with faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11 not found: ID does not exist" containerID="faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11" Nov 04 01:52:12.486555 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.486484 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11"} err="failed to get container status \"faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11\": rpc error: code = NotFound desc = could not find container \"faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11\": container with ID starting with faeb6c6d3d28d4d01c6b08830a09cfcb730aff9c90f129dad155f8648f5cdf11 not found: ID does not exist" Nov 04 01:52:12.623023 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.622987 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54"] Nov 04 01:52:12.627449 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.627420 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4955ec16-d4fd44bff-5db54"] Nov 04 01:52:12.730960 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.730845 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf"] Nov 04 01:52:12.734955 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.734928 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4955ec16-bbfc946c8-tdzxf"] Nov 04 01:52:12.748942 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.748909 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz"] Nov 04 01:52:12.749159 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:12.749107 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" podUID="25946ee4-d876-4e68-ba76-319c840636fc" containerName="maas-api" containerID="cri-o://adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7" gracePeriod=30 Nov 04 01:52:13.022704 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.022675 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:52:13.035941 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.035911 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-metrics-certs\") pod \"25946ee4-d876-4e68-ba76-319c840636fc\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " Nov 04 01:52:13.036068 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.035956 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-maas-api-tls\") pod \"25946ee4-d876-4e68-ba76-319c840636fc\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " Nov 04 01:52:13.036068 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.035989 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25946ee4-d876-4e68-ba76-319c840636fc-kube-api-access-vxmx2\" (UniqueName: \"kubernetes.io/projected/25946ee4-d876-4e68-ba76-319c840636fc-kube-api-access-vxmx2\") pod \"25946ee4-d876-4e68-ba76-319c840636fc\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " Nov 04 01:52:13.036193 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.036102 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25946ee4-d876-4e68-ba76-319c840636fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25946ee4-d876-4e68-ba76-319c840636fc-openshift-service-ca\") pod \"25946ee4-d876-4e68-ba76-319c840636fc\" (UID: \"25946ee4-d876-4e68-ba76-319c840636fc\") " Nov 04 01:52:13.036505 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.036476 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25946ee4-d876-4e68-ba76-319c840636fc-openshift-service-ca" pod "25946ee4-d876-4e68-ba76-319c840636fc" (UID: "25946ee4-d876-4e68-ba76-319c840636fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:52:13.038154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.038108 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-maas-api-tls" pod "25946ee4-d876-4e68-ba76-319c840636fc" (UID: "25946ee4-d876-4e68-ba76-319c840636fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:52:13.038154 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.038149 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-metrics-certs" pod "25946ee4-d876-4e68-ba76-319c840636fc" (UID: "25946ee4-d876-4e68-ba76-319c840636fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:52:13.038282 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.038167 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25946ee4-d876-4e68-ba76-319c840636fc-kube-api-access-vxmx2" pod "25946ee4-d876-4e68-ba76-319c840636fc" (UID: "25946ee4-d876-4e68-ba76-319c840636fc"). InnerVolumeSpecName "kube-api-access-vxmx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:52:13.137018 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.136966 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:13.137222 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.137033 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25946ee4-d876-4e68-ba76-319c840636fc-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:13.137222 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.137050 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxmx2\" (UniqueName: \"kubernetes.io/projected/25946ee4-d876-4e68-ba76-319c840636fc-kube-api-access-vxmx2\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:13.137222 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.137061 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25946ee4-d876-4e68-ba76-319c840636fc-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:13.482425 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.482379 5786 generic.go:356] "Generic (PLEG): container finished" podID="25946ee4-d876-4e68-ba76-319c840636fc" containerID="adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7" exitCode=0 Nov 04 01:52:13.482633 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.482444 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" Nov 04 01:52:13.482633 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.482455 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" event={"ID":"25946ee4-d876-4e68-ba76-319c840636fc","Type":"ContainerDied","Data":"adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7"} Nov 04 01:52:13.482633 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.482545 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz" event={"ID":"25946ee4-d876-4e68-ba76-319c840636fc","Type":"ContainerDied","Data":"591300220beb59ac24a177c5a6652af576cbcbdf8e4ce30c6dc86d445a067fbd"} Nov 04 01:52:13.482633 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.482574 5786 scope.go:122] "RemoveContainer" containerID="adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7" Nov 04 01:52:13.493205 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.493185 5786 scope.go:122] "RemoveContainer" containerID="adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7" Nov 04 01:52:13.493523 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:52:13.493503 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7\": container with ID starting with adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7 not found: ID does not exist" containerID="adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7" Nov 04 01:52:13.493571 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.493533 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7"} err="failed to get container status \"adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7\": rpc error: code = NotFound desc = could not find container \"adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7\": container with ID starting with adbb985cbf89c0d30849d3077db3e0af01633818e6d6d4e7a95ad7b9fad49be7 not found: ID does not exist" Nov 04 01:52:13.646254 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.646217 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz"] Nov 04 01:52:13.646448 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:13.646333 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4955ec16-57b5db96dc-wjfzz"] Nov 04 01:52:14.504016 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:14.503980 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cada49e-dbf8-48b4-8e66-9bd5e9512721" path="/var/lib/kubelet/pods/0cada49e-dbf8-48b4-8e66-9bd5e9512721/volumes" Nov 04 01:52:14.504444 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:14.504416 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25946ee4-d876-4e68-ba76-319c840636fc" path="/var/lib/kubelet/pods/25946ee4-d876-4e68-ba76-319c840636fc/volumes" Nov 04 01:52:14.504740 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:14.504727 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae97c0b-66e3-43cf-b6ef-03bb6e80455f" path="/var/lib/kubelet/pods/3ae97c0b-66e3-43cf-b6ef-03bb6e80455f/volumes" Nov 04 01:52:15.086273 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:52:15.086235 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:52:19.356734 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:19.356697 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:52:26.471976 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.471939 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg"] Nov 04 01:52:26.472394 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.472378 5786 cpu_manager.go:414] "containerMap: removing container" podUID="25946ee4-d876-4e68-ba76-319c840636fc" containerName="maas-api" Nov 04 01:52:26.472394 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.472389 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25946ee4-d876-4e68-ba76-319c840636fc" containerName="maas-api" Nov 04 01:52:26.472470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.472413 5786 cpu_manager.go:414] "containerMap: removing container" podUID="0cada49e-dbf8-48b4-8e66-9bd5e9512721" containerName="payload-processing" Nov 04 01:52:26.472470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.472418 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cada49e-dbf8-48b4-8e66-9bd5e9512721" containerName="payload-processing" Nov 04 01:52:26.472470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.472429 5786 cpu_manager.go:414] "containerMap: removing container" podUID="d85d60f5-b598-4909-bffd-b383252a2be5" containerName="maas-api" Nov 04 01:52:26.472470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.472434 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85d60f5-b598-4909-bffd-b383252a2be5" containerName="maas-api" Nov 04 01:52:26.472470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.472445 5786 cpu_manager.go:414] "containerMap: removing container" podUID="3ae97c0b-66e3-43cf-b6ef-03bb6e80455f" containerName="payload-pre-processing" Nov 04 01:52:26.472470 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.472449 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae97c0b-66e3-43cf-b6ef-03bb6e80455f" containerName="payload-pre-processing" Nov 04 01:52:26.472667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.472511 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cada49e-dbf8-48b4-8e66-9bd5e9512721" containerName="payload-processing" Nov 04 01:52:26.472667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.472519 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85d60f5-b598-4909-bffd-b383252a2be5" containerName="maas-api" Nov 04 01:52:26.472667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.472525 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae97c0b-66e3-43cf-b6ef-03bb6e80455f" containerName="payload-pre-processing" Nov 04 01:52:26.472667 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.472534 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="25946ee4-d876-4e68-ba76-319c840636fc" containerName="maas-api" Nov 04 01:52:26.476900 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.476879 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" Nov 04 01:52:26.576931 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.576900 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-3e5e0c-dockercfg-f899z\"" Nov 04 01:52:26.579600 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.579566 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg"] Nov 04 01:52:26.579761 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.579735 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-3e5e0c\"" Nov 04 01:52:26.585855 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.585815 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmt7\" (UniqueName: \"kubernetes.io/projected/e9e878ea-059c-4733-9c3b-51f50284cab6-kube-api-access-xwmt7\") pod \"payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg\" (UID: \"e9e878ea-059c-4733-9c3b-51f50284cab6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" Nov 04 01:52:26.586044 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.585918 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e878ea-059c-4733-9c3b-51f50284cab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg\" (UID: \"e9e878ea-059c-4733-9c3b-51f50284cab6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" Nov 04 01:52:26.626276 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.626242 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5"] Nov 04 01:52:26.629824 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.629804 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" Nov 04 01:52:26.675452 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.675419 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5"] Nov 04 01:52:26.686830 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.686795 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e878ea-059c-4733-9c3b-51f50284cab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg\" (UID: \"e9e878ea-059c-4733-9c3b-51f50284cab6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" Nov 04 01:52:26.686830 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.686834 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnvl\" (UniqueName: \"kubernetes.io/projected/79814716-f475-4987-a116-480fff82cffd-kube-api-access-2dnvl\") pod \"payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5\" (UID: \"79814716-f475-4987-a116-480fff82cffd\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" Nov 04 01:52:26.687032 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.686920 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmt7\" (UniqueName: \"kubernetes.io/projected/e9e878ea-059c-4733-9c3b-51f50284cab6-kube-api-access-xwmt7\") pod \"payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg\" (UID: \"e9e878ea-059c-4733-9c3b-51f50284cab6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" Nov 04 01:52:26.687032 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.686995 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79814716-f475-4987-a116-480fff82cffd-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5\" (UID: \"79814716-f475-4987-a116-480fff82cffd\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" Nov 04 01:52:26.687570 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.687550 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e878ea-059c-4733-9c3b-51f50284cab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg\" (UID: \"e9e878ea-059c-4733-9c3b-51f50284cab6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" Nov 04 01:52:26.742325 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.742186 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmt7\" (UniqueName: \"kubernetes.io/projected/e9e878ea-059c-4733-9c3b-51f50284cab6-kube-api-access-xwmt7\") pod \"payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg\" (UID: \"e9e878ea-059c-4733-9c3b-51f50284cab6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" Nov 04 01:52:26.788440 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.788402 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79814716-f475-4987-a116-480fff82cffd-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5\" (UID: \"79814716-f475-4987-a116-480fff82cffd\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" Nov 04 01:52:26.788600 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.788461 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnvl\" (UniqueName: \"kubernetes.io/projected/79814716-f475-4987-a116-480fff82cffd-kube-api-access-2dnvl\") pod \"payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5\" (UID: \"79814716-f475-4987-a116-480fff82cffd\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" Nov 04 01:52:26.789161 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.789141 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79814716-f475-4987-a116-480fff82cffd-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5\" (UID: \"79814716-f475-4987-a116-480fff82cffd\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" Nov 04 01:52:26.790260 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.790244 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" Nov 04 01:52:26.844597 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.843888 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnvl\" (UniqueName: \"kubernetes.io/projected/79814716-f475-4987-a116-480fff82cffd-kube-api-access-2dnvl\") pod \"payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5\" (UID: \"79814716-f475-4987-a116-480fff82cffd\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" Nov 04 01:52:26.943081 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:26.943043 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" Nov 04 01:52:27.158715 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:27.158680 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg"] Nov 04 01:52:27.164719 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:52:27.164691 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e878ea_059c_4733_9c3b_51f50284cab6.slice/crio-bf2cef9a13555b715c0fc8ea9f2da54e754187835610e7a0cd7498fdb74b9f61 WatchSource:0}: Error finding container bf2cef9a13555b715c0fc8ea9f2da54e754187835610e7a0cd7498fdb74b9f61: Status 404 returned error can't find the container with id bf2cef9a13555b715c0fc8ea9f2da54e754187835610e7a0cd7498fdb74b9f61 Nov 04 01:52:27.310970 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:27.310887 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5"] Nov 04 01:52:27.317070 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:52:27.317041 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79814716_f475_4987_a116_480fff82cffd.slice/crio-ef1b5491f51f31d342ac99d420cc3b20771c2715afc3e7278d513438327d12ad WatchSource:0}: Error finding container ef1b5491f51f31d342ac99d420cc3b20771c2715afc3e7278d513438327d12ad: Status 404 returned error can't find the container with id ef1b5491f51f31d342ac99d420cc3b20771c2715afc3e7278d513438327d12ad Nov 04 01:52:27.575180 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:27.575070 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" event={"ID":"e9e878ea-059c-4733-9c3b-51f50284cab6","Type":"ContainerStarted","Data":"d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f"} Nov 04 01:52:27.575180 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:27.575164 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" event={"ID":"e9e878ea-059c-4733-9c3b-51f50284cab6","Type":"ContainerStarted","Data":"bf2cef9a13555b715c0fc8ea9f2da54e754187835610e7a0cd7498fdb74b9f61"} Nov 04 01:52:27.575687 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:27.575197 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" Nov 04 01:52:27.576693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:27.576670 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" event={"ID":"79814716-f475-4987-a116-480fff82cffd","Type":"ContainerStarted","Data":"5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a"} Nov 04 01:52:27.576806 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:27.576715 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" event={"ID":"79814716-f475-4987-a116-480fff82cffd","Type":"ContainerStarted","Data":"ef1b5491f51f31d342ac99d420cc3b20771c2715afc3e7278d513438327d12ad"} Nov 04 01:52:27.576806 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:27.576759 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" Nov 04 01:52:27.685531 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:27.685477 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" podStartSLOduration=1.68546371 podStartE2EDuration="1.68546371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:52:27.684353283 +0000 UTC m=+3293.300699045" watchObservedRunningTime="2026-11-04 01:52:27.68546371 +0000 UTC m=+3293.301809473" Nov 04 01:52:27.796041 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:27.794706 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" podStartSLOduration=1.7946852629999999 podStartE2EDuration="1.794685263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:52:27.793502622 +0000 UTC m=+3293.409848387" watchObservedRunningTime="2026-11-04 01:52:27.794685263 +0000 UTC m=+3293.411031027" Nov 04 01:52:28.311494 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.311454 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f"] Nov 04 01:52:28.315174 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.315151 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:28.364963 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.364930 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-3e5e0c-serving-cert\"" Nov 04 01:52:28.410391 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.410357 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:28.410583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.410413 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scr2v\" (UniqueName: \"kubernetes.io/projected/06ee2c5a-6d4e-476f-af5a-757caef4f968-kube-api-access-scr2v\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:28.410583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.410530 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ee2c5a-6d4e-476f-af5a-757caef4f968-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:28.410677 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.410606 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-metrics-certs\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:28.414070 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.414049 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f"] Nov 04 01:52:28.511633 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.511587 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-metrics-certs\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:28.511816 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.511692 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:28.511816 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.511721 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scr2v\" (UniqueName: \"kubernetes.io/projected/06ee2c5a-6d4e-476f-af5a-757caef4f968-kube-api-access-scr2v\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:28.511816 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.511754 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ee2c5a-6d4e-476f-af5a-757caef4f968-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:28.511936 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:52:28.511821 5786 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-serving-cert: secret "maas-api-e2e-authcfg-http-3e5e0c-serving-cert" not found Nov 04 01:52:28.511936 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:52:28.511897 5786 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-maas-api-tls podName:06ee2c5a-6d4e-476f-af5a-757caef4f968 nodeName:}" failed. No retries permitted until 2026-11-04 01:52:29.01187683 +0000 UTC m=+3294.628222575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-maas-api-tls") pod "maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" (UID: "06ee2c5a-6d4e-476f-af5a-757caef4f968") : secret "maas-api-e2e-authcfg-http-3e5e0c-serving-cert" not found Nov 04 01:52:28.512459 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.512437 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ee2c5a-6d4e-476f-af5a-757caef4f968-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:28.514082 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.514062 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-metrics-certs\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:28.572572 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:28.572477 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scr2v\" (UniqueName: \"kubernetes.io/projected/06ee2c5a-6d4e-476f-af5a-757caef4f968-kube-api-access-scr2v\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:29.016715 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:29.016673 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:29.019307 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:29.019279 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:29.228354 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:29.228318 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:29.597840 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:29.597804 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f"] Nov 04 01:52:29.604176 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:52:29.604139 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ee2c5a_6d4e_476f_af5a_757caef4f968.slice/crio-7f76ba113688ddcc0967e75dcc7552c9a033366a89cd967f68185047920c7932 WatchSource:0}: Error finding container 7f76ba113688ddcc0967e75dcc7552c9a033366a89cd967f68185047920c7932: Status 404 returned error can't find the container with id 7f76ba113688ddcc0967e75dcc7552c9a033366a89cd967f68185047920c7932 Nov 04 01:52:30.593294 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:30.593249 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" event={"ID":"06ee2c5a-6d4e-476f-af5a-757caef4f968","Type":"ContainerStarted","Data":"7f76ba113688ddcc0967e75dcc7552c9a033366a89cd967f68185047920c7932"} Nov 04 01:52:31.598947 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:31.598912 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" event={"ID":"06ee2c5a-6d4e-476f-af5a-757caef4f968","Type":"ContainerStarted","Data":"e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da"} Nov 04 01:52:31.599388 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:31.599043 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:34.916305 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:34.916273 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:52:34.984778 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:34.984755 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:52:36.429572 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:36.429503 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" podStartSLOduration=7.497525311 podStartE2EDuration="8.429489311s" totalImagesPullingTime="931.964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:52:31.712523832 +0000 UTC m=+3297.328869597" watchObservedRunningTime="2026-11-04 01:52:36.429489311 +0000 UTC m=+3302.045835083" Nov 04 01:52:36.430431 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:36.430405 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:52:37.609510 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:37.609483 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:52:38.584836 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:38.584804 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" Nov 04 01:52:38.585061 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:52:38.584886 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" Nov 04 01:52:45.086931 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:52:45.086889 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:53:15.087400 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:53:15.087314 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:53:45.086997 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:53:45.086960 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:54:15.087311 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:54:15.087272 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:54:25.104108 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:25.104070 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5"] Nov 04 01:54:25.208499 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:25.208456 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg"] Nov 04 01:54:30.104766 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.104711 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" podUID="79814716-f475-4987-a116-480fff82cffd" containerName="payload-processing" containerID="cri-o://5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a" gracePeriod=25 Nov 04 01:54:30.209220 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.209173 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" podUID="e9e878ea-059c-4733-9c3b-51f50284cab6" containerName="payload-pre-processing" containerID="cri-o://d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f" gracePeriod=25 Nov 04 01:54:30.356472 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.356071 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" Nov 04 01:54:30.425168 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.425107 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79814716-f475-4987-a116-480fff82cffd-kube-api-access-2dnvl\" (UniqueName: \"kubernetes.io/projected/79814716-f475-4987-a116-480fff82cffd-kube-api-access-2dnvl\") pod \"79814716-f475-4987-a116-480fff82cffd\" (UID: \"79814716-f475-4987-a116-480fff82cffd\") " Nov 04 01:54:30.425362 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.425189 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79814716-f475-4987-a116-480fff82cffd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79814716-f475-4987-a116-480fff82cffd-plugins-config-volume\") pod \"79814716-f475-4987-a116-480fff82cffd\" (UID: \"79814716-f475-4987-a116-480fff82cffd\") " Nov 04 01:54:30.425592 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.425561 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79814716-f475-4987-a116-480fff82cffd-plugins-config-volume" pod "79814716-f475-4987-a116-480fff82cffd" (UID: "79814716-f475-4987-a116-480fff82cffd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:54:30.427372 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.427343 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79814716-f475-4987-a116-480fff82cffd-kube-api-access-2dnvl" pod "79814716-f475-4987-a116-480fff82cffd" (UID: "79814716-f475-4987-a116-480fff82cffd"). InnerVolumeSpecName "kube-api-access-2dnvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:54:30.466826 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.466800 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" Nov 04 01:54:30.525837 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.525807 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e878ea-059c-4733-9c3b-51f50284cab6-kube-api-access-xwmt7\" (UniqueName: \"kubernetes.io/projected/e9e878ea-059c-4733-9c3b-51f50284cab6-kube-api-access-xwmt7\") pod \"e9e878ea-059c-4733-9c3b-51f50284cab6\" (UID: \"e9e878ea-059c-4733-9c3b-51f50284cab6\") " Nov 04 01:54:30.525986 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.525893 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e878ea-059c-4733-9c3b-51f50284cab6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e878ea-059c-4733-9c3b-51f50284cab6-plugins-config-volume\") pod \"e9e878ea-059c-4733-9c3b-51f50284cab6\" (UID: \"e9e878ea-059c-4733-9c3b-51f50284cab6\") " Nov 04 01:54:30.526238 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.526222 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dnvl\" (UniqueName: \"kubernetes.io/projected/79814716-f475-4987-a116-480fff82cffd-kube-api-access-2dnvl\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:30.526289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.526235 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e878ea-059c-4733-9c3b-51f50284cab6-plugins-config-volume" pod "e9e878ea-059c-4733-9c3b-51f50284cab6" (UID: "e9e878ea-059c-4733-9c3b-51f50284cab6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:54:30.526289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.526240 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79814716-f475-4987-a116-480fff82cffd-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:30.527965 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.527943 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e878ea-059c-4733-9c3b-51f50284cab6-kube-api-access-xwmt7" pod "e9e878ea-059c-4733-9c3b-51f50284cab6" (UID: "e9e878ea-059c-4733-9c3b-51f50284cab6"). InnerVolumeSpecName "kube-api-access-xwmt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:54:30.627092 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.626999 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwmt7\" (UniqueName: \"kubernetes.io/projected/e9e878ea-059c-4733-9c3b-51f50284cab6-kube-api-access-xwmt7\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:30.627092 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:30.627036 5786 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e878ea-059c-4733-9c3b-51f50284cab6-plugins-config-volume\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:31.168224 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.168186 5786 generic.go:356] "Generic (PLEG): container finished" podID="e9e878ea-059c-4733-9c3b-51f50284cab6" containerID="d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f" exitCode=0 Nov 04 01:54:31.168693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.168242 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" Nov 04 01:54:31.168693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.168277 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" event={"ID":"e9e878ea-059c-4733-9c3b-51f50284cab6","Type":"ContainerDied","Data":"d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f"} Nov 04 01:54:31.168693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.168336 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg" event={"ID":"e9e878ea-059c-4733-9c3b-51f50284cab6","Type":"ContainerDied","Data":"bf2cef9a13555b715c0fc8ea9f2da54e754187835610e7a0cd7498fdb74b9f61"} Nov 04 01:54:31.168693 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.168370 5786 scope.go:122] "RemoveContainer" containerID="d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f" Nov 04 01:54:31.169720 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.169604 5786 generic.go:356] "Generic (PLEG): container finished" podID="79814716-f475-4987-a116-480fff82cffd" containerID="5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a" exitCode=0 Nov 04 01:54:31.169720 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.169653 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" event={"ID":"79814716-f475-4987-a116-480fff82cffd","Type":"ContainerDied","Data":"5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a"} Nov 04 01:54:31.169720 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.169661 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" Nov 04 01:54:31.169720 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.169686 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5" event={"ID":"79814716-f475-4987-a116-480fff82cffd","Type":"ContainerDied","Data":"ef1b5491f51f31d342ac99d420cc3b20771c2715afc3e7278d513438327d12ad"} Nov 04 01:54:31.185457 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.185435 5786 scope.go:122] "RemoveContainer" containerID="d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f" Nov 04 01:54:31.185762 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:54:31.185745 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f\": container with ID starting with d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f not found: ID does not exist" containerID="d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f" Nov 04 01:54:31.185812 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.185769 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f"} err="failed to get container status \"d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f\": rpc error: code = NotFound desc = could not find container \"d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f\": container with ID starting with d5c5aa23d8d89c09e4c4158a522cda6bf297606b81aa91915577e244d2f3f79f not found: ID does not exist" Nov 04 01:54:31.185812 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.185786 5786 scope.go:122] "RemoveContainer" containerID="5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a" Nov 04 01:54:31.193999 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.193974 5786 scope.go:122] "RemoveContainer" containerID="5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a" Nov 04 01:54:31.194344 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:54:31.194319 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a\": container with ID starting with 5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a not found: ID does not exist" containerID="5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a" Nov 04 01:54:31.194454 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.194358 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a"} err="failed to get container status \"5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a\": rpc error: code = NotFound desc = could not find container \"5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a\": container with ID starting with 5f064b82e4b22e2ea8f675cffaba2c4b0dc1e59e79fd7554e1d5c1f6a7cb813a not found: ID does not exist" Nov 04 01:54:31.325021 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.324991 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg"] Nov 04 01:54:31.372600 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.372515 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-3e5e0c-d97489495-mj8sg"] Nov 04 01:54:31.436343 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.436312 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5"] Nov 04 01:54:31.440427 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:31.440401 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-3e5e0c-65fc7bb454-h9vm5"] Nov 04 01:54:32.505008 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:32.504976 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79814716-f475-4987-a116-480fff82cffd" path="/var/lib/kubelet/pods/79814716-f475-4987-a116-480fff82cffd/volumes" Nov 04 01:54:32.505424 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:32.505401 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e878ea-059c-4733-9c3b-51f50284cab6" path="/var/lib/kubelet/pods/e9e878ea-059c-4733-9c3b-51f50284cab6/volumes" Nov 04 01:54:33.549223 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:33.549189 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f"] Nov 04 01:54:33.549617 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:33.549389 5786 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" podUID="06ee2c5a-6d4e-476f-af5a-757caef4f968" containerName="maas-api" containerID="cri-o://e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da" gracePeriod=30 Nov 04 01:54:33.824375 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:33.824351 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:54:33.957374 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:33.957343 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-metrics-certs\") pod \"06ee2c5a-6d4e-476f-af5a-757caef4f968\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " Nov 04 01:54:33.957548 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:33.957407 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ee2c5a-6d4e-476f-af5a-757caef4f968-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ee2c5a-6d4e-476f-af5a-757caef4f968-openshift-service-ca\") pod \"06ee2c5a-6d4e-476f-af5a-757caef4f968\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " Nov 04 01:54:33.957548 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:33.957516 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-maas-api-tls\") pod \"06ee2c5a-6d4e-476f-af5a-757caef4f968\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " Nov 04 01:54:33.957692 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:33.957667 5786 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ee2c5a-6d4e-476f-af5a-757caef4f968-kube-api-access-scr2v\" (UniqueName: \"kubernetes.io/projected/06ee2c5a-6d4e-476f-af5a-757caef4f968-kube-api-access-scr2v\") pod \"06ee2c5a-6d4e-476f-af5a-757caef4f968\" (UID: \"06ee2c5a-6d4e-476f-af5a-757caef4f968\") " Nov 04 01:54:33.957798 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:33.957776 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ee2c5a-6d4e-476f-af5a-757caef4f968-openshift-service-ca" pod "06ee2c5a-6d4e-476f-af5a-757caef4f968" (UID: "06ee2c5a-6d4e-476f-af5a-757caef4f968"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:54:33.958058 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:33.958032 5786 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ee2c5a-6d4e-476f-af5a-757caef4f968-openshift-service-ca\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:33.959609 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:33.959583 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-metrics-certs" pod "06ee2c5a-6d4e-476f-af5a-757caef4f968" (UID: "06ee2c5a-6d4e-476f-af5a-757caef4f968"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:54:33.959696 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:33.959631 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ee2c5a-6d4e-476f-af5a-757caef4f968-kube-api-access-scr2v" pod "06ee2c5a-6d4e-476f-af5a-757caef4f968" (UID: "06ee2c5a-6d4e-476f-af5a-757caef4f968"). InnerVolumeSpecName "kube-api-access-scr2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:54:33.959696 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:33.959634 5786 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-maas-api-tls" pod "06ee2c5a-6d4e-476f-af5a-757caef4f968" (UID: "06ee2c5a-6d4e-476f-af5a-757caef4f968"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:54:34.062870 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.062774 5786 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-maas-api-tls\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:34.063135 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.063079 5786 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scr2v\" (UniqueName: \"kubernetes.io/projected/06ee2c5a-6d4e-476f-af5a-757caef4f968-kube-api-access-scr2v\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:34.063135 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.063111 5786 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ee2c5a-6d4e-476f-af5a-757caef4f968-metrics-certs\") on node \"ip-10-0-136-94.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:34.188333 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.188268 5786 generic.go:356] "Generic (PLEG): container finished" podID="06ee2c5a-6d4e-476f-af5a-757caef4f968" containerID="e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da" exitCode=0 Nov 04 01:54:34.188333 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.188315 5786 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" Nov 04 01:54:34.188584 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.188347 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" event={"ID":"06ee2c5a-6d4e-476f-af5a-757caef4f968","Type":"ContainerDied","Data":"e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da"} Nov 04 01:54:34.188584 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.188406 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f" event={"ID":"06ee2c5a-6d4e-476f-af5a-757caef4f968","Type":"ContainerDied","Data":"7f76ba113688ddcc0967e75dcc7552c9a033366a89cd967f68185047920c7932"} Nov 04 01:54:34.188584 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.188435 5786 scope.go:122] "RemoveContainer" containerID="e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da" Nov 04 01:54:34.200380 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.200361 5786 scope.go:122] "RemoveContainer" containerID="e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da" Nov 04 01:54:34.200689 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:54:34.200665 5786 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da\": container with ID starting with e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da not found: ID does not exist" containerID="e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da" Nov 04 01:54:34.200733 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.200697 5786 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da"} err="failed to get container status \"e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da\": rpc error: code = NotFound desc = could not find container \"e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da\": container with ID starting with e9c2226501993033631c7d8c7d5f957d054583819c46fb802d96a59a04fb38da not found: ID does not exist" Nov 04 01:54:34.344851 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.344822 5786 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f"] Nov 04 01:54:34.392825 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.392792 5786 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3e5e0c-7778bc5987-tlh5f"] Nov 04 01:54:34.504873 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:34.504838 5786 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ee2c5a-6d4e-476f-af5a-757caef4f968" path="/var/lib/kubelet/pods/06ee2c5a-6d4e-476f-af5a-757caef4f968/volumes" Nov 04 01:54:38.232688 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:38.232654 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:54:45.086156 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:54:45.086062 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:54:51.956187 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:54:51.956154 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:55:04.558908 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:55:04.558875 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:55:15.086708 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:55:15.086667 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:55:25.641505 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:55:25.641465 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:55:45.086991 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:55:45.086948 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:55:58.162767 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:55:58.162732 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6j2lc"] Nov 04 01:56:09.862359 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:09.862327 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-566cd577c-j4xcj_00f9fdff-f532-47b4-ae77-6cf7e35aa000/manager/0.log" Nov 04 01:56:10.182165 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:10.182048 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mg6l2_d6608682-17b7-425a-bca8-55c10a066b86/server/0.log" Nov 04 01:56:10.397884 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:10.397853 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-746fj_e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78/manager/0.log" Nov 04 01:56:11.435998 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:11.435958 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6fc6d86564-2p9h5_63a4db80-e943-4e66-9990-2d07ecdcbfa7/maas-api/0.log" Nov 04 01:56:15.086799 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:56:15.086718 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:56:15.732088 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:15.732060 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6j2lc_65e12552-a3fa-4259-9b61-52b7f27d6b03/limitador/0.log" Nov 04 01:56:17.039796 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:17.039762 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-cjfd5_85d1e41d-fded-429d-99b6-e941ffc740a6/discovery/0.log" Nov 04 01:56:17.286901 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:17.286873 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5794fb467c-v8xjm_8b1540a2-97cc-4956-836f-cdf0c8f8d810/kube-auth-proxy/0.log" Nov 04 01:56:18.810825 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:18.810798 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-ct688_a47f6c57-b397-4190-b510-bab0db9443a3/istio-proxy/0.log" Nov 04 01:56:19.563811 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:19.563785 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mjcm9_2434f011-3d55-42fa-8a2d-b07df64862ef/istio-proxy/0.log" Nov 04 01:56:19.800534 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:19.800506 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-qvl5w_d802bf36-fe5d-46cf-ab84-208d75861871/istio-proxy/0.log" Nov 04 01:56:20.043594 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:20.043568 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-rhjlh_543f2b63-4c53-4e48-b7e7-048c472336df/istio-proxy/0.log" Nov 04 01:56:20.971003 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:20.970948 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-767db4f785-j87sp_28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/router/0.log" Nov 04 01:56:22.085039 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:22.085010 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-85cpd_3a5a107e-19f3-4611-90ad-f345b54e27f5/main/0.log" Nov 04 01:56:22.616840 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:22.616813 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-hz9wk_5701df54-b91d-44ca-b43c-de2454ab59d8/main/0.log" Nov 04 01:56:22.847847 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:22.847816 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k_d0506411-64e5-47a9-b8b2-d03f78db8d67/main/0.log" Nov 04 01:56:23.544967 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:23.544937 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2_0b8dfc4b-7e71-4360-91cd-a22c7685da9e/main/0.log" Nov 04 01:56:35.106327 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:35.106294 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-566cd577c-j4xcj_00f9fdff-f532-47b4-ae77-6cf7e35aa000/manager/0.log" Nov 04 01:56:35.385479 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:35.385391 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mg6l2_d6608682-17b7-425a-bca8-55c10a066b86/server/0.log" Nov 04 01:56:35.618568 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:35.618537 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-746fj_e2bfe1fc-5272-43b7-ba5b-e57f2c8e9d78/manager/0.log" Nov 04 01:56:36.445278 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:36.445246 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6fc6d86564-2p9h5_63a4db80-e943-4e66-9990-2d07ecdcbfa7/maas-api/0.log" Nov 04 01:56:40.316057 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:40.316027 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6j2lc_65e12552-a3fa-4259-9b61-52b7f27d6b03/limitador/0.log" Nov 04 01:56:41.390538 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:41.390509 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-cjfd5_85d1e41d-fded-429d-99b6-e941ffc740a6/discovery/0.log" Nov 04 01:56:41.613888 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:41.613859 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5794fb467c-v8xjm_8b1540a2-97cc-4956-836f-cdf0c8f8d810/kube-auth-proxy/0.log" Nov 04 01:56:43.199858 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:43.199822 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-ct688_a47f6c57-b397-4190-b510-bab0db9443a3/istio-proxy/0.log" Nov 04 01:56:43.911097 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:43.911052 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mjcm9_2434f011-3d55-42fa-8a2d-b07df64862ef/istio-proxy/0.log" Nov 04 01:56:44.148494 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:44.148462 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-qvl5w_d802bf36-fe5d-46cf-ab84-208d75861871/istio-proxy/0.log" Nov 04 01:56:44.361158 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:44.361114 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-rhjlh_543f2b63-4c53-4e48-b7e7-048c472336df/istio-proxy/0.log" Nov 04 01:56:45.086897 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:56:45.086857 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:56:45.264794 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:45.264758 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-767db4f785-j87sp_28aa6fa1-33f8-450e-a1f1-d382e7cf6a47/router/0.log" Nov 04 01:56:46.103871 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:46.103825 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-85cpd_3a5a107e-19f3-4611-90ad-f345b54e27f5/main/0.log" Nov 04 01:56:46.557289 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:46.557192 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-hz9wk_5701df54-b91d-44ca-b43c-de2454ab59d8/main/0.log" Nov 04 01:56:46.772927 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:46.772870 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kvh5k_d0506411-64e5-47a9-b8b2-d03f78db8d67/main/0.log" Nov 04 01:56:47.476440 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:56:47.476404 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-v62m2_0b8dfc4b-7e71-4360-91cd-a22c7685da9e/main/0.log" Nov 04 01:57:15.086251 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:57:15.086212 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:57:17.255815 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:57:17.255782 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-566cd577c-j4xcj_00f9fdff-f532-47b4-ae77-6cf7e35aa000/manager/0.log" Nov 04 01:57:17.846436 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:57:17.846401 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-566cd577c-j4xcj_00f9fdff-f532-47b4-ae77-6cf7e35aa000/manager/0.log" Nov 04 01:57:34.946579 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:57:34.946549 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:57:35.015248 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:57:35.015219 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-478s5_e2534417-f751-4241-b22d-3ea465dc6a19/console-operator/2.log" Nov 04 01:57:45.087623 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:57:45.087542 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:57:56.926583 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:57:56.926548 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-566cd577c-j4xcj_00f9fdff-f532-47b4-ae77-6cf7e35aa000/manager/0.log" Nov 04 01:57:57.514389 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:57:57.514338 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-566cd577c-j4xcj_00f9fdff-f532-47b4-ae77-6cf7e35aa000/manager/0.log" Nov 04 01:58:10.982195 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:10.982165 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pclxc_0ebe7f81-60bb-423b-a871-c2bab48db5c3/global-pull-secret-syncer/0.log" Nov 04 01:58:12.241758 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:12.241729 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bnf5w_593868da-9895-4d98-9da1-4fa815c11674/connection-checker/0.log" Nov 04 01:58:12.691291 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:12.691259 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-gwvrx_b098c99f-60e8-40bd-a431-fa252a476a0b/connection-checker/0.log" Nov 04 01:58:13.143638 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:13.143612 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-knfg8_81991817-9603-48e1-b25f-0392e58a83b5/connection-checker/0.log" Nov 04 01:58:14.271247 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:14.271206 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7smhm_d15630f9-042e-4c31-8f70-c8733c9dc31e/konnectivity-agent/0.log" Nov 04 01:58:15.087397 ip-10-0-136-94 kubenswrapper[5786]: E1104 01:58:15.087356 5786 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w6x6k" containerName="extractor" Nov 04 01:58:16.062975 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:16.062916 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-94.us-west-2.compute.internal_48bdaec858a4ad8070f76d6d06532a70/haproxy/0.log" Nov 04 01:58:19.425645 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.425607 5786 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69"] Nov 04 01:58:19.426063 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.426001 5786 cpu_manager.go:414] "containerMap: removing container" podUID="06ee2c5a-6d4e-476f-af5a-757caef4f968" containerName="maas-api" Nov 04 01:58:19.426063 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.426011 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ee2c5a-6d4e-476f-af5a-757caef4f968" containerName="maas-api" Nov 04 01:58:19.426063 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.426028 5786 cpu_manager.go:414] "containerMap: removing container" podUID="79814716-f475-4987-a116-480fff82cffd" containerName="payload-processing" Nov 04 01:58:19.426063 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.426033 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79814716-f475-4987-a116-480fff82cffd" containerName="payload-processing" Nov 04 01:58:19.426063 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.426060 5786 cpu_manager.go:414] "containerMap: removing container" podUID="e9e878ea-059c-4733-9c3b-51f50284cab6" containerName="payload-pre-processing" Nov 04 01:58:19.426269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.426066 5786 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e878ea-059c-4733-9c3b-51f50284cab6" containerName="payload-pre-processing" Nov 04 01:58:19.426269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.426136 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e878ea-059c-4733-9c3b-51f50284cab6" containerName="payload-pre-processing" Nov 04 01:58:19.426269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.426148 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ee2c5a-6d4e-476f-af5a-757caef4f968" containerName="maas-api" Nov 04 01:58:19.426269 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.426155 5786 memory_manager.go:365] "RemoveStaleState removing state" podUID="79814716-f475-4987-a116-480fff82cffd" containerName="payload-processing" Nov 04 01:58:19.429213 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.429190 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.483203 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.483170 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7gzq6\"/\"openshift-service-ca.crt\"" Nov 04 01:58:19.483391 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.483204 5786 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7gzq6\"/\"kube-root-ca.crt\"" Nov 04 01:58:19.486618 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.486589 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hsjl\" (UniqueName: \"kubernetes.io/projected/0e75d2d4-c27d-43db-887c-5497a053dda4-kube-api-access-6hsjl\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.486758 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.486627 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e75d2d4-c27d-43db-887c-5497a053dda4-proc\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.486758 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.486647 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e75d2d4-c27d-43db-887c-5497a053dda4-podres\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.486758 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.486702 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e75d2d4-c27d-43db-887c-5497a053dda4-lib-modules\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.486869 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.486772 5786 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e75d2d4-c27d-43db-887c-5497a053dda4-sys\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.532400 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.532365 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69"] Nov 04 01:58:19.578931 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.578900 5786 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7gzq6\"/\"default-dockercfg-pd6hx\"" Nov 04 01:58:19.587387 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.587355 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hsjl\" (UniqueName: \"kubernetes.io/projected/0e75d2d4-c27d-43db-887c-5497a053dda4-kube-api-access-6hsjl\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.587526 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.587478 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e75d2d4-c27d-43db-887c-5497a053dda4-proc\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.587590 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.587523 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e75d2d4-c27d-43db-887c-5497a053dda4-podres\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.587590 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.587564 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e75d2d4-c27d-43db-887c-5497a053dda4-lib-modules\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.587695 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.587601 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e75d2d4-c27d-43db-887c-5497a053dda4-proc\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.587695 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.587668 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e75d2d4-c27d-43db-887c-5497a053dda4-podres\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.587814 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.587706 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e75d2d4-c27d-43db-887c-5497a053dda4-lib-modules\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.587814 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.587740 5786 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e75d2d4-c27d-43db-887c-5497a053dda4-sys\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.587898 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.587809 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e75d2d4-c27d-43db-887c-5497a053dda4-sys\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.643282 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.643248 5786 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hsjl\" (UniqueName: \"kubernetes.io/projected/0e75d2d4-c27d-43db-887c-5497a053dda4-kube-api-access-6hsjl\") pod \"perf-node-gather-daemonset-lrs69\" (UID: \"0e75d2d4-c27d-43db-887c-5497a053dda4\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:19.743713 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:19.743680 5786 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:20.112957 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:20.112922 5786 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69"] Nov 04 01:58:20.118238 ip-10-0-136-94 kubenswrapper[5786]: W1104 01:58:20.118208 5786 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0e75d2d4_c27d_43db_887c_5497a053dda4.slice/crio-ec18d204e872f7abff58777a886c9eab5212c7d24b43345409a1a13be524eea0 WatchSource:0}: Error finding container ec18d204e872f7abff58777a886c9eab5212c7d24b43345409a1a13be524eea0: Status 404 returned error can't find the container with id ec18d204e872f7abff58777a886c9eab5212c7d24b43345409a1a13be524eea0 Nov 04 01:58:20.334669 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:20.334633 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" event={"ID":"0e75d2d4-c27d-43db-887c-5497a053dda4","Type":"ContainerStarted","Data":"93000294ba741e69706754902ceba68f35e1f5fd37a3f96d236c5fd49bd7dc1f"} Nov 04 01:58:20.334879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:20.334721 5786 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" event={"ID":"0e75d2d4-c27d-43db-887c-5497a053dda4","Type":"ContainerStarted","Data":"ec18d204e872f7abff58777a886c9eab5212c7d24b43345409a1a13be524eea0"} Nov 04 01:58:20.334879 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:20.334835 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:20.446616 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:20.446500 5786 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" podStartSLOduration=1.446485956 podStartE2EDuration="1.446485956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:58:20.444762685 +0000 UTC m=+3646.061108449" watchObservedRunningTime="2026-11-04 01:58:20.446485956 +0000 UTC m=+3646.062831848" Nov 04 01:58:21.303372 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:21.303335 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6j2lc_65e12552-a3fa-4259-9b61-52b7f27d6b03/limitador/0.log" Nov 04 01:58:26.350331 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:26.350301 5786 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-lrs69" Nov 04 01:58:27.729026 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:27.728990 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-4bql4_21952e92-3601-4969-adbd-6c6ff9b87074/cluster-monitoring-operator/0.log" Nov 04 01:58:33.177670 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:33.177639 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rxt2c_8ec4a7cb-76bf-4c22-8141-04417bc3a515/node-exporter/0.log" Nov 04 01:58:33.621652 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:33.621624 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rxt2c_8ec4a7cb-76bf-4c22-8141-04417bc3a515/kube-rbac-proxy/0.log" Nov 04 01:58:34.063370 ip-10-0-136-94 kubenswrapper[5786]: I1104 01:58:34.063295 5786 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rxt2c_8ec4a7cb-76bf-4c22-8141-04417bc3a515/init-textfile/0.log"