Nov 03 06:58:23.616224 ip-10-0-138-121 systemd[1]: Starting Kubernetes Kubelet... Nov 03 06:58:23.990563 ip-10-0-138-121 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 06:58:23.990563 ip-10-0-138-121 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 06:58:23.990563 ip-10-0-138-121 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 06:58:23.990563 ip-10-0-138-121 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 06:58:23.993720 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993628 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:23.993720 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993721 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993725 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993729 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993732 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993734 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993738 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993740 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993743 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993745 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993748 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993751 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993753 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993756 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993759 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993761 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993764 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993766 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993776 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993779 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993782 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:23.993789 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993784 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993787 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993790 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993793 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993796 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993798 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993801 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993804 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993806 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993809 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993811 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993814 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993816 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993819 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993821 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993823 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993826 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993829 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993832 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993834 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:23.994266 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993837 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993839 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993841 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993845 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993849 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993852 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993855 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993857 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993860 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993863 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993865 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993867 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993870 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993873 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993875 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993878 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993881 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993883 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993886 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:23.994764 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993888 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993890 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993893 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993895 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993898 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993900 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993903 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993905 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993907 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993910 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993912 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993915 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993922 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993925 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993941 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993946 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993950 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993954 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993957 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:23.995224 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993960 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993963 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993966 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993969 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993972 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993975 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993977 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993980 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993982 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993985 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993987 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993990 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993992 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993995 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.993997 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994000 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994002 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994005 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994008 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994017 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:23.995671 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994020 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994024 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994027 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994029 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994032 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994034 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994037 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994774 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994781 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994783 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994787 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994791 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994794 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994797 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994800 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994803 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994806 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994809 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994812 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:23.996172 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994814 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994817 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994819 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994822 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994825 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994827 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994829 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994832 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994834 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994837 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994839 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994842 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994844 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994847 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994849 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994852 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994855 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994857 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994859 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994862 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:23.996625 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994865 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994867 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994870 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994872 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994875 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994877 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994880 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994882 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994885 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994888 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994891 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994894 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994896 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994899 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994901 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994904 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994906 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994909 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994912 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:23.997101 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994914 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994917 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994919 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994922 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994924 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994941 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994944 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994946 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994949 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994952 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994954 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994956 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994959 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994962 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994964 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994967 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994970 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994972 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994975 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:23.997547 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994977 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994980 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994982 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994985 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994988 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994991 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994994 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994996 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.994999 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995001 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995004 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995006 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995008 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995011 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995014 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995016 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995019 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995021 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995024 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:23.998000 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995027 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995029 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995031 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995034 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995036 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995039 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995041 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995044 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995046 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995049 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995052 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995055 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995057 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995060 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995064 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995068 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995071 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995532 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995537 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995540 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:23.998473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995543 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995546 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995549 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995551 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995554 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995556 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995559 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995562 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995564 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995567 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995569 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995572 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995574 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995577 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995580 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995582 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995585 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995587 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995590 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995593 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:23.999590 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995595 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995600 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995603 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995605 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995608 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995610 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995613 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995616 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995618 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995620 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995623 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995626 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995628 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995631 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995633 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995636 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995638 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995640 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995643 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995645 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:24.000066 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995648 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995651 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995653 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995656 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995658 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995661 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995663 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995666 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995668 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995671 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995674 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995677 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995679 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995682 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995685 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995687 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995690 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995693 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995695 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995698 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:24.000580 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995700 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995702 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995705 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995707 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995710 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995713 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995716 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995719 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995723 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995726 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995728 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995731 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995733 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995735 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995738 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995740 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995742 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995745 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995747 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:24.001078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995750 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995752 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995755 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995757 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995760 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995762 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995765 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995768 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995770 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995772 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995775 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995777 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995780 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995782 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995784 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995787 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995789 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995791 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995794 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:24.001509 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995797 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995799 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995804 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995808 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.995811 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997527 2575 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997536 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997542 2575 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997547 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997552 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997556 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997561 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997565 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997569 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997571 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997575 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997579 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997582 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997585 2575 flags.go:64] FLAG: --cgroup-root="" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997588 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997591 2575 flags.go:64] FLAG: --client-ca-file="" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997593 2575 flags.go:64] FLAG: --cloud-provider="external" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997596 2575 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 06:58:24.002019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997601 2575 flags.go:64] FLAG: --cluster-domain="" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997604 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997607 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997609 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997613 2575 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997617 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997625 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997628 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997632 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997635 2575 flags.go:64] FLAG: --contention-profiling="false" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997639 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997642 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997645 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997648 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997652 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997655 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997658 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997661 2575 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997664 2575 flags.go:64] FLAG: --enable-server="true" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997667 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997671 2575 flags.go:64] FLAG: --event-burst="100" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997674 2575 flags.go:64] FLAG: --event-qps="50" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997677 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997680 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 06:58:24.002587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997683 2575 flags.go:64] FLAG: --eviction-hard="" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997687 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997689 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997692 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997695 2575 flags.go:64] FLAG: --eviction-soft="" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997698 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997701 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997704 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997707 2575 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997709 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997712 2575 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997715 2575 flags.go:64] FLAG: --feature-gates="" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997719 2575 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997722 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997726 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997729 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997733 2575 flags.go:64] FLAG: --healthz-port="10248" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997736 2575 flags.go:64] FLAG: --help="false" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997739 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997742 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997745 2575 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997748 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997751 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 06:58:24.003199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997755 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997757 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997760 2575 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997763 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997766 2575 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997769 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997772 2575 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997774 2575 flags.go:64] FLAG: --kube-reserved="" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997777 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997781 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997785 2575 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997787 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997790 2575 flags.go:64] FLAG: --lock-file="" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997793 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997796 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997799 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997804 2575 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997807 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997810 2575 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997813 2575 flags.go:64] FLAG: --logging-format="text" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997816 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997819 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997822 2575 flags.go:64] FLAG: --manifest-url="" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997826 2575 flags.go:64] FLAG: --manifest-url-header="" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997830 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 06:58:24.003759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997833 2575 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997837 2575 flags.go:64] FLAG: --max-pods="110" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997841 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997844 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997847 2575 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997850 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997853 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997855 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997858 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997865 2575 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997868 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997871 2575 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997874 2575 flags.go:64] FLAG: --pod-cidr="" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997877 2575 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997880 2575 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997883 2575 flags.go:64] FLAG: --pods-per-core="0" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997887 2575 flags.go:64] FLAG: --port="10250" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997890 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997893 2575 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-062409b512d164cbf" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997896 2575 flags.go:64] FLAG: --qos-reserved="" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997899 2575 flags.go:64] FLAG: --read-only-port="10255" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997902 2575 flags.go:64] FLAG: --register-node="true" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997905 2575 flags.go:64] FLAG: --register-with-taints="" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997909 2575 flags.go:64] FLAG: --registry-burst="10" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997912 2575 flags.go:64] FLAG: --registry-qps="5" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997914 2575 flags.go:64] FLAG: --reserved-cpus="" Nov 03 06:58:24.004366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997917 2575 flags.go:64] FLAG: --reserved-memory="" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997921 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997924 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997939 2575 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997944 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997947 2575 flags.go:64] FLAG: --runonce="false" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997950 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997953 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997956 2575 flags.go:64] FLAG: --seccomp-default="false" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997959 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997962 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997965 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997969 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997972 2575 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997975 2575 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997978 2575 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997980 2575 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997983 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997986 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997989 2575 flags.go:64] FLAG: --system-cgroups="" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997992 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.997997 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998002 2575 flags.go:64] FLAG: --tls-cert-file="" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998004 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998016 2575 flags.go:64] FLAG: --tls-min-version="" Nov 03 06:58:24.004987 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998020 2575 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998023 2575 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998025 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998028 2575 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998031 2575 flags.go:64] FLAG: --v="2" Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998036 2575 flags.go:64] FLAG: --version="false" Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998039 2575 flags.go:64] FLAG: --vmodule="" Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998043 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998046 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998196 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998201 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998205 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998208 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998210 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998213 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998216 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998219 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998222 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998232 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998237 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998240 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:24.005877 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998243 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998246 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998248 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998251 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998254 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998256 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998259 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998261 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998265 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998268 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998271 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998273 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998276 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998278 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998281 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998283 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998285 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998288 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998290 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998293 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:24.006454 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998295 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998298 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998302 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998304 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998307 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998309 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998312 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998314 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998316 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998319 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998327 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998330 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998332 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998335 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998338 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998342 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998345 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998348 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998350 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:24.006980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998352 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998358 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998360 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998363 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998365 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998368 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998370 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998373 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998375 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998378 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998381 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998383 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998386 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998388 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998391 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998400 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998403 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998405 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998408 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998411 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:24.007521 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998413 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998416 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998418 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998421 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998430 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998433 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998436 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998438 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998440 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998443 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998445 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998448 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998450 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998454 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998457 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998459 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998462 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998464 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998467 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:24.008021 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998469 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998472 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998474 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998477 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998480 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998482 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998485 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998487 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998490 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998493 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998496 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998498 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998500 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998503 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998505 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998508 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:24.008473 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:23.998534 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998583 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998593 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998596 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998599 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998601 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998604 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998606 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998609 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998611 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998614 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998617 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998619 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998622 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998624 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998627 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998629 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998631 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998634 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998636 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:24.008851 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998639 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998641 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998643 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998646 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998648 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998651 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998654 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998656 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998658 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998661 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998663 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998665 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998668 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998670 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998673 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998680 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998683 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998686 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998688 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:24.009336 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998690 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998693 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998695 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998697 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998700 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998702 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998704 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998707 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998709 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998711 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998714 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998716 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998718 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998720 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998723 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998725 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998727 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998730 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998732 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998734 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:24.009782 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998737 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998739 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998741 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998744 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998747 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998750 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998753 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998756 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998758 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998766 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998769 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998771 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998774 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998777 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998779 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998781 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998784 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998787 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998789 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:24.010267 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998791 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998793 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998796 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998798 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998800 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998803 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998805 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998807 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998810 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998812 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998815 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998817 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998819 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998822 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998824 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998826 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998829 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998831 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998833 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:24.010708 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998836 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998838 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998840 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998845 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998855 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998858 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998860 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998863 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998865 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:23.998868 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.007891 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.007906 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.008644 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.008651 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008712 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008718 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008721 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008723 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008726 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:24.011190 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008729 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008731 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008733 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008736 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008739 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008742 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008744 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008747 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008749 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008752 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008754 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008756 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008759 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008762 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008764 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008767 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008770 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008772 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008774 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:24.011624 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008778 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008783 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008785 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008788 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008790 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008793 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008795 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008798 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008800 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008803 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008805 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008807 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008810 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008812 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008815 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008817 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008820 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008823 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008825 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008828 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:24.012166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008830 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008833 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008836 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008839 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008841 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008843 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008846 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008850 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008854 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008857 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008860 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008863 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008865 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008868 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008871 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008873 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008876 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008878 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008881 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:24.012631 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008883 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008885 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008888 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008890 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008892 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008895 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008897 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008899 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008902 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008904 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008906 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008909 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008911 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008914 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008916 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008918 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008921 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008924 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008939 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:24.013078 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008941 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008944 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008947 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008949 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008952 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008954 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008957 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008959 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008962 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008964 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008967 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008969 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008972 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008975 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008977 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008980 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008982 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008984 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008987 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008989 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:24.013514 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008992 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008994 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008996 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.008999 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.009026 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009163 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009168 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009171 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009173 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009176 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009179 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009181 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009184 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009186 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009189 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009191 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:24.013980 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009194 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009197 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009200 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009202 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009204 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009207 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009209 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009212 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009214 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009216 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009219 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009221 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009224 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009226 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009228 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009231 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009233 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009235 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009238 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009240 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:24.014366 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009243 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009245 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009248 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009250 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009253 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009255 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009257 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009261 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009264 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009266 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009268 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009271 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009273 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009275 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009278 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009280 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009282 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009285 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009287 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009290 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:24.014835 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009292 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009294 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009297 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009299 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009302 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009304 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009308 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009312 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009314 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009317 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009320 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009323 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009325 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009328 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009330 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009333 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009335 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009337 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009340 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:24.015298 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009343 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009346 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009350 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009353 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009358 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009361 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009363 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009365 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009368 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009370 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009373 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009375 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009377 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009380 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009382 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009385 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009387 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009389 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009392 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:24.015738 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009394 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009396 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009399 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009401 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009403 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009406 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009408 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009410 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009413 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009415 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009418 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009420 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009422 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009424 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009427 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009430 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:24.016307 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009432 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.009456 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009487 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009491 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009495 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009497 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009500 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009503 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009506 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009509 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009512 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009514 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009517 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009520 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009523 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009527 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:58:24.016678 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009530 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009533 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009536 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009539 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009541 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009543 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009546 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009548 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009550 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009553 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009556 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009558 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009560 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009564 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009568 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009571 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009574 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009578 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009580 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:58:24.017062 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009583 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009585 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009588 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009590 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009592 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009595 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009597 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009600 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009602 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009605 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009607 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009609 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009612 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009614 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009617 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009619 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009621 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009624 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009626 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009629 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:58:24.017501 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009631 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009633 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009636 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009638 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009640 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009643 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009645 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009647 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009650 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009652 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009656 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009658 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009661 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009663 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009666 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009668 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009670 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009673 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009675 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:58:24.017981 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009678 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009680 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009683 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009685 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009687 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009690 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009692 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009695 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009698 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009700 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009703 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009705 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009707 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009710 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009712 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009715 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009717 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009719 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009722 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009724 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:58:24.018426 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009727 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009730 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009733 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009737 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009740 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009743 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009745 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009749 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009751 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009754 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009756 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009759 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009761 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:24.009764 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.009819 2575 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.014114 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.015048 2575 server.go:1014] "Starting client certificate rotation" Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.015143 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 06:58:24.018923 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.015178 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 06:58:24.019357 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.017071 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 06:58:24.019357 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.018125 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 06:58:24.028064 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.028046 2575 log.go:25] "Validated CRI v1 runtime API" Nov 03 06:58:24.031835 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.031818 2575 log.go:25] "Validated CRI v1 image API" Nov 03 06:58:24.032650 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.032629 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 06:58:24.035227 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.035205 2575 fs.go:135] Filesystem UUIDs: map[2a91b4d3-e463-4a40-9ccf-c5458756fa5b:/dev/nvme0n1p3 72604b0a-4219-40dc-9a63-63e626dd03ef:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 06:58:24.035298 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.035228 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 06:58:24.039859 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.039757 2575 manager.go:217] Machine: {Timestamp:2026-11-03 06:58:24.039299648 +0000 UTC m=+0.309059899 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104269 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2df219d8404368faa465956e378006 SystemUUID:ec2df219-d840-4368-faa4-65956e378006 BootID:06df53c6-48e5-4686-b627-66894f6f5cd5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:01:88:bf:2b:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:01:88:bf:2b:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:b9:3b:39:f9:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 06:58:24.039859 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.039856 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 06:58:24.039980 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.039968 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 06:58:24.042400 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.042337 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 06:58:24.042544 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.042401 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-121.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 06:58:24.042596 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.042546 2575 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 06:58:24.042596 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.042555 2575 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 06:58:24.042596 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.042573 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 06:58:24.044079 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.044068 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 06:58:24.045233 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.045223 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 06:58:24.049544 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.049532 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 06:58:24.049675 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.049665 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 06:58:24.051275 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.051249 2575 kubelet.go:493] "Attempting to sync node with API server" Nov 03 06:58:24.051381 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.051287 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 06:58:24.051381 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.051303 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 06:58:24.051381 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.051319 2575 kubelet.go:405] "Adding apiserver pod source" Nov 03 06:58:24.051381 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.051331 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 06:58:24.051856 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.051830 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 06:58:24.054613 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.054597 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 06:58:24.060966 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.060954 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 06:58:24.061839 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.061828 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 06:58:24.061876 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.061857 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 06:58:24.062030 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062020 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 06:58:24.062065 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062035 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 06:58:24.062065 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062041 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 06:58:24.062065 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062046 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 06:58:24.062065 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062052 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 06:58:24.062065 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062057 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 06:58:24.062811 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062803 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 06:58:24.062842 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062812 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 06:58:24.062842 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062819 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 06:58:24.062842 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062824 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 06:58:24.062842 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062832 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 06:58:24.062842 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062841 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 06:58:24.063106 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062865 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 06:58:24.063106 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.062871 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 06:58:24.066592 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.066579 2575 server.go:1263] "Started kubelet" Nov 03 06:58:24.066714 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.066674 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 06:58:24.066776 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.066692 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 06:58:24.066776 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.066764 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 06:58:24.067249 ip-10-0-138-121 systemd[1]: Started Kubernetes Kubelet. Nov 03 06:58:24.069022 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.069003 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 06:58:24.069955 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.069924 2575 server.go:317] "Adding debug handlers to kubelet server" Nov 03 06:58:24.073041 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:24.073021 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 06:58:24.076098 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.076083 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 06:58:24.076575 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.076556 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 06:58:24.076696 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:24.076678 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-121.us-west-1.compute.internal\" not found" Nov 03 06:58:24.077243 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.077226 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 06:58:24.077243 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.077230 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 06:58:24.077395 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.077256 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 06:58:24.077532 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.077517 2575 reconstruct.go:97] "Volume reconstruction finished" Nov 03 06:58:24.077585 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.077532 2575 reconciler.go:29] "Reconciler: start to sync state" Nov 03 06:58:24.077585 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.077576 2575 factory.go:55] Registering systemd factory Nov 03 06:58:24.077649 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.077588 2575 factory.go:223] Registration of the systemd container factory successfully Nov 03 06:58:24.077774 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.077760 2575 factory.go:153] Registering CRI-O factory Nov 03 06:58:24.077826 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.077775 2575 factory.go:223] Registration of the crio container factory successfully Nov 03 06:58:24.077826 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.077814 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 06:58:24.077826 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.077826 2575 factory.go:103] Registering Raw factory Nov 03 06:58:24.077980 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.077837 2575 manager.go:1199] Started watching for new ooms in manager Nov 03 06:58:24.078966 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.078946 2575 manager.go:319] Starting recovery of all containers Nov 03 06:58:24.089902 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.089884 2575 manager.go:324] Recovery completed Nov 03 06:58:24.094224 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.094205 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:24.095636 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.095607 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:24.095709 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.095653 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:24.095709 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.095667 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:24.096243 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.096228 2575 cpu_manager.go:226] "Starting" policy="none" Nov 03 06:58:24.096243 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.096238 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:24.096367 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.096240 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 06:58:24.096367 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.096310 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 06:58:24.096874 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.096856 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:24.096998 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.096891 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:24.096998 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.096908 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:24.098443 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.098425 2575 policy_none.go:50] "Start" Nov 03 06:58:24.098443 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.098440 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 06:58:24.098570 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.098449 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 06:58:24.100355 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.100342 2575 policy_none.go:44] "Start" Nov 03 06:58:24.144587 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.144459 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 06:58:24.148373 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.148359 2575 manager.go:341] "Starting Device Plugin manager" Nov 03 06:58:24.148461 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:24.148391 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 06:58:24.148461 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.148403 2575 server.go:87] "Starting device plugin registration server" Nov 03 06:58:24.148641 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.148628 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 06:58:24.148694 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.148642 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 06:58:24.148740 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.148711 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 06:58:24.148819 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.148805 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 06:58:24.148819 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.148818 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 06:58:24.149284 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:24.149264 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 06:58:24.149343 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:24.149306 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-121.us-west-1.compute.internal\" not found" Nov 03 06:58:24.197579 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.197558 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 06:58:24.199278 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.198873 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 06:58:24.199278 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.198892 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 06:58:24.199278 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.198909 2575 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 06:58:24.199278 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:24.198960 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 06:58:24.212064 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.212046 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-121.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 06:58:24.212862 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:24.212041 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-121.us-west-1.compute.internal.18e476bc9d602642 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-121.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-121.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 06:58:24.06655341 +0000 UTC m=+0.336313658,LastTimestamp:2026-11-03 06:58:24.06655341 +0000 UTC m=+0.336313658,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-121.us-west-1.compute.internal,}" Nov 03 06:58:24.249395 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.249346 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:24.250053 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.250038 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:24.250117 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.250069 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:24.250117 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.250082 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:24.250117 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.250108 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.299299 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.299254 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal"] Nov 03 06:58:24.299366 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.299324 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:24.300034 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.300020 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:24.300091 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.300043 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:24.300091 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.300058 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:24.302067 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.302056 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:24.302198 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.302185 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.302232 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.302212 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:24.302711 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.302696 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:24.302773 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.302723 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:24.302773 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.302737 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:24.302773 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.302745 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:24.302773 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.302761 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:24.302773 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.302773 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:24.304690 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.304673 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.304776 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.304707 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:58:24.305348 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.305327 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:58:24.305423 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.305361 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:58:24.305423 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.305373 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:58:24.306610 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.306593 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 06:58:24.306686 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.306652 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zqx4n" Nov 03 06:58:24.341391 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:24.341366 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-121.us-west-1.compute.internal\" not found" node="ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.350885 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:24.350868 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-121.us-west-1.compute.internal\" not found" node="ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.371143 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.371122 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:24.374410 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.374396 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:24.376577 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.376564 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:24.380871 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.380857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/305af933eb9c6382ff5eabda70a436a6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"305af933eb9c6382ff5eabda70a436a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.380915 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.380879 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c99460852de5d8046fca947f507d3971-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"c99460852de5d8046fca947f507d3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.380915 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.380896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c99460852de5d8046fca947f507d3971-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"c99460852de5d8046fca947f507d3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.420728 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.420708 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.420787 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:24.420733 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-121.us-west-1.compute.internal\": node \"ip-10-0-138-121.us-west-1.compute.internal\" not found" Nov 03 06:58:24.471019 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.470999 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:24.477027 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.477010 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.481769 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.481746 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/305af933eb9c6382ff5eabda70a436a6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"305af933eb9c6382ff5eabda70a436a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.481869 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.481771 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c99460852de5d8046fca947f507d3971-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"c99460852de5d8046fca947f507d3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.481869 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.481789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c99460852de5d8046fca947f507d3971-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"c99460852de5d8046fca947f507d3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.481869 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.481834 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c99460852de5d8046fca947f507d3971-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"c99460852de5d8046fca947f507d3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.481869 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.481851 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/305af933eb9c6382ff5eabda70a436a6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"305af933eb9c6382ff5eabda70a436a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.482038 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.481866 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c99460852de5d8046fca947f507d3971-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"c99460852de5d8046fca947f507d3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.540963 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.540916 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 06:58:24.541652 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.541641 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.603876 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.603859 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 06:58:24.644719 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.644697 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:24.653395 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:24.653379 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:58:25.014721 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.014640 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 06:58:25.015256 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.014763 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 06:58:25.015256 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.014820 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 06:58:25.015256 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.014837 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 06:58:25.015256 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.014829 2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa5d097cf21004010b37820ad9a97310-f5a8f4c20a3cd5e4.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-121.us-west-1.compute.internal.18e476bc9f1bea8e\": read tcp 10.0.138.121:60290->13.184.88.160:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-121.us-west-1.compute.internal.18e476bc9f1bea8e default 5025 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-121.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-121.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-121.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 06:58:24 +0000 UTC,LastTimestamp:2026-11-03 06:58:24.300032738 +0000 UTC m=+0.569792986,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-121.us-west-1.compute.internal,}" Nov 03 06:58:25.052413 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.052394 2575 apiserver.go:52] "Watching apiserver" Nov 03 06:58:25.076339 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.076321 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 06:58:25.310563 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.310470 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 06:53:25 +0000 UTC" deadline="2028-05-11 19:04:48.202558305 +0000 UTC" Nov 03 06:58:25.310563 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.310518 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13332h6m22.892044245s" Nov 03 06:58:25.328972 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.328948 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 06:58:25.329254 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.329238 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v","openshift-multus/multus-additional-cni-plugins-4ndld","openshift-multus/multus-rww5p","openshift-multus/network-metrics-daemon-cd7lc","openshift-cluster-node-tuning-operator/tuned-6f246","openshift-image-registry/node-ca-nn9v4","openshift-ovn-kubernetes/ovnkube-node-l4ffc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-zhpcz","kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal","openshift-dns/node-resolver-mwws4","openshift-network-diagnostics/network-check-target-9gjvc","openshift-network-operator/iptables-alerter-s5f7z","kube-system/konnectivity-agent-nj9qb"] Nov 03 06:58:25.331522 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.331506 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.333607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.333587 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.335604 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.335583 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.337593 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.337579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:25.337662 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.337647 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:25.339566 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.339550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.341559 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.341545 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:58:25.343832 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.343815 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.345950 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.345919 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:25.346064 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.345992 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:25.348170 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.348149 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mwws4" Nov 03 06:58:25.351167 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.351152 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:25.351244 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.351217 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:25.353741 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.353726 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:58:25.356464 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.356449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:58:25.378504 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.378487 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 06:58:25.386329 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovnkube-script-lib\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.386400 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-run\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.386400 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg99r\" (UniqueName: \"kubernetes.io/projected/46730579-83ec-4a31-8a10-759a76d63eac-kube-api-access-wg99r\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.386400 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-log-socket\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.386400 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovnkube-config\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.386584 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386407 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-tmp-dir\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:58:25.386584 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386434 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-kubelet\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.386584 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpzg2\" (UniqueName: \"kubernetes.io/projected/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-kube-api-access-xpzg2\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.386584 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysconfig\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.386584 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-lib-modules\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.386584 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386521 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-device-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.386584 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386535 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-env-overrides\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.386584 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaee23aa-3918-44a3-a35c-7a77133e45c7-iptables-alerter-script\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:58:25.386584 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eff68601-5319-4dc8-93c6-16de197bd812-konnectivity-ca\") pod \"konnectivity-agent-nj9qb\" (UID: \"eff68601-5319-4dc8-93c6-16de197bd812\") " pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:58:25.386584 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2z4q\" (UniqueName: \"kubernetes.io/projected/eaee23aa-3918-44a3-a35c-7a77133e45c7-kube-api-access-f2z4q\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386629 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-cni-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-modprobe-d\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysctl-d\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-sys\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386710 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-cni-netd\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386734 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxt7\" (UniqueName: \"kubernetes.io/projected/3378427f-b215-4a67-8a9d-aab1f8b05d8f-kube-api-access-txxt7\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386773 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-cnibin\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-cni-binary-copy\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386840 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386864 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-system-cni-dir\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-cnibin\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386909 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-netns\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.387021 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-cni-bin\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.386982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-ovn\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8d5f4689-033d-4aa8-8603-a56cbdd449cb-dbus\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-systemd-units\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387049 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-run-netns\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-systemd\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387135 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-host\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387205 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46730579-83ec-4a31-8a10-759a76d63eac-etc-tuned\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387228 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qk6d\" (UniqueName: \"kubernetes.io/projected/9fbdc7ee-c57c-4a65-ba85-b20706762749-kube-api-access-7qk6d\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387261 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387278 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-var-lib-kubelet\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387321 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387327 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-socket-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387351 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-sys-fs\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387365 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l9lht\"" Nov 03 06:58:25.387673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387376 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovn-node-metrics-cert\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-hosts-file\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387467 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7gmw\" (UniqueName: \"kubernetes.io/projected/3e9850e7-1c78-455a-b2d0-77971e830f80-kube-api-access-t7gmw\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-system-cni-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-hostroot\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387577 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-conf-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zx74\" (UniqueName: \"kubernetes.io/projected/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-kube-api-access-9zx74\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-cni-multus\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-multus-certs\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387677 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46730579-83ec-4a31-8a10-759a76d63eac-tmp\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fbdc7ee-c57c-4a65-ba85-b20706762749-serviceca\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387754 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8d5f4689-033d-4aa8-8603-a56cbdd449cb-kubelet-config\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387798 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.388307 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387816 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-slash\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-var-lib-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-node-log\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387869 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaee23aa-3918-44a3-a35c-7a77133e45c7-host-slash\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eff68601-5319-4dc8-93c6-16de197bd812-agent-certs\") pod \"konnectivity-agent-nj9qb\" (UID: \"eff68601-5319-4dc8-93c6-16de197bd812\") " pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysctl-conf\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387939 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-systemd\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-kubelet\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq9tc\" (UniqueName: \"kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc\") pod \"network-check-target-9gjvc\" (UID: \"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3\") " pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.387984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-k8s-cni-cncf-io\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-daemon-config\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-kubernetes\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fbdc7ee-c57c-4a65-ba85-b20706762749-host\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388079 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-registration-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388112 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-cni-bin\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.388807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-etc-kubernetes\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.389282 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-socket-dir-parent\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.389282 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwk9n\" (UniqueName: \"kubernetes.io/projected/8d832341-de73-4a7b-9ce0-c41bbe655035-kube-api-access-mwk9n\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:25.389282 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388181 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxlxl\" (UniqueName: \"kubernetes.io/projected/fe326448-785b-4e7a-a189-76298b83c9b3-kube-api-access-jxlxl\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.389282 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388208 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-etc-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.389282 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388265 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44mbd\"" Nov 03 06:58:25.389282 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388745 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 06:58:25.389282 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.388793 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 06:58:25.389475 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.389426 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 06:58:25.389655 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.389644 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 06:58:25.389774 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.389763 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 06:58:25.390121 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.390106 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 06:58:25.390374 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.390360 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5znfn\"" Nov 03 06:58:25.394124 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.394110 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dmgt\"" Nov 03 06:58:25.400942 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.400918 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 06:58:25.401311 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.401297 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-br79x\"" Nov 03 06:58:25.402837 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.402796 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 06:58:25.404230 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.403787 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:58:25.404230 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.404009 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 06:58:25.436852 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.436832 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 06:58:25.437300 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.437263 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8c889\"" Nov 03 06:58:25.437300 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.437284 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 06:58:25.437419 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.437300 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-htlhr\"" Nov 03 06:58:25.437419 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.437364 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 06:58:25.437419 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.437387 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 06:58:25.437522 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.437426 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 06:58:25.438099 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.438087 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 06:58:25.438255 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.438242 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 06:58:25.439105 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.439084 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 06:58:25.439146 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.439132 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 06:58:25.439301 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.439289 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 06:58:25.439696 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.439679 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:58:25.439797 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.439783 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 06:58:25.440441 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.440426 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lr4sn\"" Nov 03 06:58:25.441096 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.441064 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 06:58:25.445027 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.445011 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxfbc\"" Nov 03 06:58:25.445079 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.445070 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 06:58:25.445123 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.445085 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 06:58:25.445123 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.445100 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6vjh8" Nov 03 06:58:25.488421 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.488522 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovn-node-metrics-cert\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.488522 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488446 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-hosts-file\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:58:25.488522 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488503 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.488668 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488509 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-hosts-file\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:58:25.488668 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.488668 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7gmw\" (UniqueName: \"kubernetes.io/projected/3e9850e7-1c78-455a-b2d0-77971e830f80-kube-api-access-t7gmw\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.488668 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488580 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-system-cni-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.488668 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488604 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-hostroot\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.488668 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-conf-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.488668 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488639 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-system-cni-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.488668 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zx74\" (UniqueName: \"kubernetes.io/projected/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-kube-api-access-9zx74\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488679 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-hostroot\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-cni-multus\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-multus-certs\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488724 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-conf-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488724 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488763 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-cni-multus\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488773 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-multus-certs\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488794 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46730579-83ec-4a31-8a10-759a76d63eac-tmp\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fbdc7ee-c57c-4a65-ba85-b20706762749-serviceca\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8d5f4689-033d-4aa8-8603-a56cbdd449cb-kubelet-config\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488962 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.488993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-slash\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489016 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-var-lib-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489024 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8d5f4689-033d-4aa8-8603-a56cbdd449cb-kubelet-config\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:25.489041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-node-log\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaee23aa-3918-44a3-a35c-7a77133e45c7-host-slash\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489084 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-slash\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489019 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489114 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaee23aa-3918-44a3-a35c-7a77133e45c7-host-slash\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489115 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-var-lib-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489136 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eff68601-5319-4dc8-93c6-16de197bd812-agent-certs\") pod \"konnectivity-agent-nj9qb\" (UID: \"eff68601-5319-4dc8-93c6-16de197bd812\") " pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysctl-conf\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489191 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489135 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-node-log\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-systemd\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-kubelet\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489293 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489318 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysctl-conf\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489318 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fbdc7ee-c57c-4a65-ba85-b20706762749-serviceca\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9tc\" (UniqueName: \"kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc\") pod \"network-check-target-9gjvc\" (UID: \"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3\") " pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489367 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-kubelet\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489374 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-systemd\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.489755 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-k8s-cni-cncf-io\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-daemon-config\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-kubernetes\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489492 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-k8s-cni-cncf-io\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489507 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-kubernetes\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fbdc7ee-c57c-4a65-ba85-b20706762749-host\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489551 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fbdc7ee-c57c-4a65-ba85-b20706762749-host\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-registration-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-cni-bin\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489617 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-etc-kubernetes\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-socket-dir-parent\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.489644 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489660 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-registration-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489674 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-cni-bin\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489709 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk9n\" (UniqueName: \"kubernetes.io/projected/8d832341-de73-4a7b-9ce0-c41bbe655035-kube-api-access-mwk9n\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489724 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-etc-kubernetes\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlxl\" (UniqueName: \"kubernetes.io/projected/fe326448-785b-4e7a-a189-76298b83c9b3-kube-api-access-jxlxl\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.490607 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489768 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-socket-dir-parent\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.489781 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret podName:8d5f4689-033d-4aa8-8603-a56cbdd449cb nodeName:}" failed. No retries permitted until 2026-11-03 06:58:25.989743251 +0000 UTC m=+2.259503495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret") pod "global-pull-secret-syncer-zhpcz" (UID: "8d5f4689-033d-4aa8-8603-a56cbdd449cb") : object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-etc-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovnkube-script-lib\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-run\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489883 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-etc-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489913 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-run\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489960 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-daemon-config\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.489970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg99r\" (UniqueName: \"kubernetes.io/projected/46730579-83ec-4a31-8a10-759a76d63eac-kube-api-access-wg99r\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490192 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-log-socket\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovnkube-config\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-tmp-dir\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490283 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-log-socket\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490319 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-kubelet\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpzg2\" (UniqueName: \"kubernetes.io/projected/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-kube-api-access-xpzg2\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490375 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysconfig\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490398 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-lib-modules\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.491438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-device-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-env-overrides\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490478 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaee23aa-3918-44a3-a35c-7a77133e45c7-iptables-alerter-script\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eff68601-5319-4dc8-93c6-16de197bd812-konnectivity-ca\") pod \"konnectivity-agent-nj9qb\" (UID: \"eff68601-5319-4dc8-93c6-16de197bd812\") " pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490526 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490542 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-tmp-dir\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2z4q\" (UniqueName: \"kubernetes.io/projected/eaee23aa-3918-44a3-a35c-7a77133e45c7-kube-api-access-f2z4q\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490624 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-cni-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490659 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovnkube-script-lib\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-modprobe-d\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysctl-d\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-sys\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490751 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-modprobe-d\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-cni-netd\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490789 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-kubelet\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.492268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxt7\" (UniqueName: \"kubernetes.io/projected/3378427f-b215-4a67-8a9d-aab1f8b05d8f-kube-api-access-txxt7\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490822 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490850 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-cnibin\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-cni-binary-copy\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490918 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysctl-d\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-system-cni-dir\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-cnibin\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.490989 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-sys\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-netns\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-cni-bin\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491027 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-cni-netd\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491134 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-cni-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491154 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491274 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovnkube-config\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491389 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysconfig\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491401 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-system-cni-dir\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.492882 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491413 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-device-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491469 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491508 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-netns\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491523 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-cnibin\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491558 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-cnibin\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491602 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-cni-bin\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491837 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491877 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-ovn\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491905 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8d5f4689-033d-4aa8-8603-a56cbdd449cb-dbus\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.491987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-systemd-units\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-run-netns\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492036 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-env-overrides\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492063 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaee23aa-3918-44a3-a35c-7a77133e45c7-iptables-alerter-script\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492073 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-cni-binary-copy\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492080 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-ovn\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492087 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-systemd\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492040 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-systemd\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492125 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-run-netns\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.493448 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492177 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-lib-modules\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.492186 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-host\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46730579-83ec-4a31-8a10-759a76d63eac-etc-tuned\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.492228 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs podName:8d832341-de73-4a7b-9ce0-c41bbe655035 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:25.992213323 +0000 UTC m=+2.261973579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs") pod "network-metrics-daemon-cd7lc" (UID: "8d832341-de73-4a7b-9ce0-c41bbe655035") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qk6d\" (UniqueName: \"kubernetes.io/projected/9fbdc7ee-c57c-4a65-ba85-b20706762749-kube-api-access-7qk6d\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492301 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-var-lib-kubelet\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492305 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-host\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492322 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-socket-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492343 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-sys-fs\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492409 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-sys-fs\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492569 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-var-lib-kubelet\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492568 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8d5f4689-033d-4aa8-8603-a56cbdd449cb-dbus\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.492611 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-systemd-units\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.493242 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovn-node-metrics-cert\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.494008 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.493568 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-socket-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.494537 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.493606 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46730579-83ec-4a31-8a10-759a76d63eac-tmp\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.494537 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.493779 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.494537 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.493842 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eff68601-5319-4dc8-93c6-16de197bd812-agent-certs\") pod \"konnectivity-agent-nj9qb\" (UID: \"eff68601-5319-4dc8-93c6-16de197bd812\") " pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:58:25.494537 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.494107 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eff68601-5319-4dc8-93c6-16de197bd812-konnectivity-ca\") pod \"konnectivity-agent-nj9qb\" (UID: \"eff68601-5319-4dc8-93c6-16de197bd812\") " pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:58:25.494537 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.494348 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46730579-83ec-4a31-8a10-759a76d63eac-etc-tuned\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.556393 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.556372 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2z4q\" (UniqueName: \"kubernetes.io/projected/eaee23aa-3918-44a3-a35c-7a77133e45c7-kube-api-access-f2z4q\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:58:25.556621 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.556606 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg99r\" (UniqueName: \"kubernetes.io/projected/46730579-83ec-4a31-8a10-759a76d63eac-kube-api-access-wg99r\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.556890 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.556865 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpzg2\" (UniqueName: \"kubernetes.io/projected/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-kube-api-access-xpzg2\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.557014 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.556997 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7gmw\" (UniqueName: \"kubernetes.io/projected/3e9850e7-1c78-455a-b2d0-77971e830f80-kube-api-access-t7gmw\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.557057 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.557019 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zx74\" (UniqueName: \"kubernetes.io/projected/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-kube-api-access-9zx74\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:58:25.557849 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.557833 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 06:58:25.557924 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.557856 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 06:58:25.557924 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.557869 2575 projected.go:196] Error preparing data for projected volume kube-api-access-fq9tc for pod openshift-network-diagnostics/network-check-target-9gjvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:25.558044 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.557953 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc podName:f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:26.057918691 +0000 UTC m=+2.327678941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fq9tc" (UniqueName: "kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc") pod "network-check-target-9gjvc" (UID: "f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:25.559748 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.559730 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwk9n\" (UniqueName: \"kubernetes.io/projected/8d832341-de73-4a7b-9ce0-c41bbe655035-kube-api-access-mwk9n\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:25.560160 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.560144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qk6d\" (UniqueName: \"kubernetes.io/projected/9fbdc7ee-c57c-4a65-ba85-b20706762749-kube-api-access-7qk6d\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:58:25.561424 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.561369 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxt7\" (UniqueName: \"kubernetes.io/projected/3378427f-b215-4a67-8a9d-aab1f8b05d8f-kube-api-access-txxt7\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.562357 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.562339 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlxl\" (UniqueName: \"kubernetes.io/projected/fe326448-785b-4e7a-a189-76298b83c9b3-kube-api-access-jxlxl\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.659179 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.659160 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:58:25.667244 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:25.667208 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99460852de5d8046fca947f507d3971.slice/crio-455ee9c4a945f69306bc3eed3f8d59f87b9c9a30a84caf8e9961bbdc5df9985e WatchSource:0}: Error finding container 455ee9c4a945f69306bc3eed3f8d59f87b9c9a30a84caf8e9961bbdc5df9985e: Status 404 returned error can't find the container with id 455ee9c4a945f69306bc3eed3f8d59f87b9c9a30a84caf8e9961bbdc5df9985e Nov 03 06:58:25.669031 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:25.669000 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe326448_785b_4e7a_a189_76298b83c9b3.slice/crio-9dd28d3f845f33093c3d5b509fef1d627c672056ce7519537f74a44b99801afc WatchSource:0}: Error finding container 9dd28d3f845f33093c3d5b509fef1d627c672056ce7519537f74a44b99801afc: Status 404 returned error can't find the container with id 9dd28d3f845f33093c3d5b509fef1d627c672056ce7519537f74a44b99801afc Nov 03 06:58:25.673098 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.673083 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 06:58:25.682163 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.682147 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rww5p" Nov 03 06:58:25.686559 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.686545 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:58:25.689759 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:25.689741 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6551e74b_f4c1_4be3_8fbf_cf5f262caea4.slice/crio-71545bab3073c0490e06e49cb45a626b739105f2371f2502405b0b990912057d WatchSource:0}: Error finding container 71545bab3073c0490e06e49cb45a626b739105f2371f2502405b0b990912057d: Status 404 returned error can't find the container with id 71545bab3073c0490e06e49cb45a626b739105f2371f2502405b0b990912057d Nov 03 06:58:25.695694 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:25.695677 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9850e7_1c78_455a_b2d0_77971e830f80.slice/crio-ebb62cd3d5a2fc7f0a61610142cf822d42be28743edc9b1e49113c2d541433ff WatchSource:0}: Error finding container ebb62cd3d5a2fc7f0a61610142cf822d42be28743edc9b1e49113c2d541433ff: Status 404 returned error can't find the container with id ebb62cd3d5a2fc7f0a61610142cf822d42be28743edc9b1e49113c2d541433ff Nov 03 06:58:25.711759 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.711743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:58:25.712856 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:25.712839 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305af933eb9c6382ff5eabda70a436a6.slice/crio-4a7637175a7288f0ee21f9233dd9d1812a75823a3a415497ee2af6f9c92b09e3 WatchSource:0}: Error finding container 4a7637175a7288f0ee21f9233dd9d1812a75823a3a415497ee2af6f9c92b09e3: Status 404 returned error can't find the container with id 4a7637175a7288f0ee21f9233dd9d1812a75823a3a415497ee2af6f9c92b09e3 Nov 03 06:58:25.719182 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:25.719160 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46730579_83ec_4a31_8a10_759a76d63eac.slice/crio-0af07c0e8a414088d580b85245fbfb790f177da422bd4ec680dde32d0414440f WatchSource:0}: Error finding container 0af07c0e8a414088d580b85245fbfb790f177da422bd4ec680dde32d0414440f: Status 404 returned error can't find the container with id 0af07c0e8a414088d580b85245fbfb790f177da422bd4ec680dde32d0414440f Nov 03 06:58:25.725773 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.725757 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:58:25.734166 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:25.734150 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbdc7ee_c57c_4a65_ba85_b20706762749.slice/crio-f3d86743aa0ad0ccb0d4f17c4e71fc42e5fe4a2865b14ce3c1d8134fd3107274 WatchSource:0}: Error finding container f3d86743aa0ad0ccb0d4f17c4e71fc42e5fe4a2865b14ce3c1d8134fd3107274: Status 404 returned error can't find the container with id f3d86743aa0ad0ccb0d4f17c4e71fc42e5fe4a2865b14ce3c1d8134fd3107274 Nov 03 06:58:25.757041 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.757018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:25.759130 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.759113 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mwws4" Nov 03 06:58:25.764886 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:25.764867 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3378427f_b215_4a67_8a9d_aab1f8b05d8f.slice/crio-c94a7ec137bf0c0fd9c279bdda4970a2afa45c008c369f16fb6800a8d32a083c WatchSource:0}: Error finding container c94a7ec137bf0c0fd9c279bdda4970a2afa45c008c369f16fb6800a8d32a083c: Status 404 returned error can't find the container with id c94a7ec137bf0c0fd9c279bdda4970a2afa45c008c369f16fb6800a8d32a083c Nov 03 06:58:25.768616 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:25.768596 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa0cc98_1f99_4197_9a87_52c9ea4277f9.slice/crio-17aad86066f1d41a3aebd4f5cf755efb5681d5be9c99e3789aeb7a6d0556a431 WatchSource:0}: Error finding container 17aad86066f1d41a3aebd4f5cf755efb5681d5be9c99e3789aeb7a6d0556a431: Status 404 returned error can't find the container with id 17aad86066f1d41a3aebd4f5cf755efb5681d5be9c99e3789aeb7a6d0556a431 Nov 03 06:58:25.771649 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.771635 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:58:25.776304 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:25.776286 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:58:25.780332 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:25.780315 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaee23aa_3918_44a3_a35c_7a77133e45c7.slice/crio-b4d96c4ee8a0304c03e8f1d2aec1f5f93f05f428394517bc860855b940456b32 WatchSource:0}: Error finding container b4d96c4ee8a0304c03e8f1d2aec1f5f93f05f428394517bc860855b940456b32: Status 404 returned error can't find the container with id b4d96c4ee8a0304c03e8f1d2aec1f5f93f05f428394517bc860855b940456b32 Nov 03 06:58:25.785045 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:25.785027 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff68601_5319_4dc8_93c6_16de197bd812.slice/crio-756aba6dcca68596c6f2fd64c7970ea1eba3f709bfff83e8987a2763ff2f12d5 WatchSource:0}: Error finding container 756aba6dcca68596c6f2fd64c7970ea1eba3f709bfff83e8987a2763ff2f12d5: Status 404 returned error can't find the container with id 756aba6dcca68596c6f2fd64c7970ea1eba3f709bfff83e8987a2763ff2f12d5 Nov 03 06:58:25.786755 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.786739 2575 kuberuntime_manager.go:1707] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-nj9qb_kube-system(eff68601-5319-4dc8-93c6-16de197bd812): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Nov 03 06:58:25.786795 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:25.786766 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ErrImagePull: \"pull QPS exceeded\"" pod="kube-system/konnectivity-agent-nj9qb" podUID="eff68601-5319-4dc8-93c6-16de197bd812" Nov 03 06:58:27.019443 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.019376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:27.019443 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.019424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:27.019612 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:27.019506 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:27.019612 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:27.019523 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:27.019612 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:27.019548 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret podName:8d5f4689-033d-4aa8-8603-a56cbdd449cb nodeName:}" failed. No retries permitted until 2026-11-03 06:58:28.019536799 +0000 UTC m=+3.265514381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret") pod "global-pull-secret-syncer-zhpcz" (UID: "8d5f4689-033d-4aa8-8603-a56cbdd449cb") : object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:27.019612 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:27.019576 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs podName:8d832341-de73-4a7b-9ce0-c41bbe655035 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:28.019559737 +0000 UTC m=+3.265537320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs") pod "network-metrics-daemon-cd7lc" (UID: "8d832341-de73-4a7b-9ce0-c41bbe655035") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:27.119985 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.119960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9tc\" (UniqueName: \"kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc\") pod \"network-check-target-9gjvc\" (UID: \"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3\") " pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:27.120550 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:27.120069 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 06:58:27.120550 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:27.120083 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 06:58:27.120550 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:27.120091 2575 projected.go:196] Error preparing data for projected volume kube-api-access-fq9tc for pod openshift-network-diagnostics/network-check-target-9gjvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:27.120550 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:27.120126 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc podName:f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:28.12011695 +0000 UTC m=+3.366094533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq9tc" (UniqueName: "kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc") pod "network-check-target-9gjvc" (UID: "f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:27.161056 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.161033 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:27.226452 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.226407 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nn9v4" event={"ID":"9fbdc7ee-c57c-4a65-ba85-b20706762749","Type":"ContainerStarted","Data":"f3d86743aa0ad0ccb0d4f17c4e71fc42e5fe4a2865b14ce3c1d8134fd3107274"} Nov 03 06:58:27.227410 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.227383 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mwws4" event={"ID":"2fa0cc98-1f99-4197-9a87-52c9ea4277f9","Type":"ContainerStarted","Data":"17aad86066f1d41a3aebd4f5cf755efb5681d5be9c99e3789aeb7a6d0556a431"} Nov 03 06:58:27.228616 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.228590 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" event={"ID":"3378427f-b215-4a67-8a9d-aab1f8b05d8f","Type":"ContainerStarted","Data":"c94a7ec137bf0c0fd9c279bdda4970a2afa45c008c369f16fb6800a8d32a083c"} Nov 03 06:58:27.229275 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.229260 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:27.229756 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.229737 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6f246" event={"ID":"46730579-83ec-4a31-8a10-759a76d63eac","Type":"ContainerStarted","Data":"0af07c0e8a414088d580b85245fbfb790f177da422bd4ec680dde32d0414440f"} Nov 03 06:58:27.230885 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.230865 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" event={"ID":"305af933eb9c6382ff5eabda70a436a6","Type":"ContainerStarted","Data":"4a7637175a7288f0ee21f9233dd9d1812a75823a3a415497ee2af6f9c92b09e3"} Nov 03 06:58:27.231940 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.231920 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ndld" event={"ID":"3e9850e7-1c78-455a-b2d0-77971e830f80","Type":"ContainerStarted","Data":"ebb62cd3d5a2fc7f0a61610142cf822d42be28743edc9b1e49113c2d541433ff"} Nov 03 06:58:27.232933 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.232916 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rww5p" event={"ID":"6551e74b-f4c1-4be3-8fbf-cf5f262caea4","Type":"ContainerStarted","Data":"71545bab3073c0490e06e49cb45a626b739105f2371f2502405b0b990912057d"} Nov 03 06:58:27.233896 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.233878 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" event={"ID":"fe326448-785b-4e7a-a189-76298b83c9b3","Type":"ContainerStarted","Data":"9dd28d3f845f33093c3d5b509fef1d627c672056ce7519537f74a44b99801afc"} Nov 03 06:58:27.234911 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.234892 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" event={"ID":"c99460852de5d8046fca947f507d3971","Type":"ContainerStarted","Data":"455ee9c4a945f69306bc3eed3f8d59f87b9c9a30a84caf8e9961bbdc5df9985e"} Nov 03 06:58:27.236014 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.235995 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nj9qb" event={"ID":"eff68601-5319-4dc8-93c6-16de197bd812","Type":"ContainerStarted","Data":"756aba6dcca68596c6f2fd64c7970ea1eba3f709bfff83e8987a2763ff2f12d5"} Nov 03 06:58:27.237170 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.237152 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s5f7z" event={"ID":"eaee23aa-3918-44a3-a35c-7a77133e45c7","Type":"ContainerStarted","Data":"b4d96c4ee8a0304c03e8f1d2aec1f5f93f05f428394517bc860855b940456b32"} Nov 03 06:58:27.237555 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:27.237521 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef3d8a5c5a290fbfb7b3688e9a224e9c47d9f3326f05a0f73577656437e3f406\\\": ErrImagePull: pull QPS exceeded\"" pod="kube-system/konnectivity-agent-nj9qb" podUID="eff68601-5319-4dc8-93c6-16de197bd812" Nov 03 06:58:27.322403 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.322352 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:58:27.472254 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.472173 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 06:53:26 +0000 UTC" deadline="2028-04-13 20:58:36.069047989 +0000 UTC" Nov 03 06:58:27.472254 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:27.472252 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12662h0m8.596804494s" Nov 03 06:58:28.028204 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:28.028166 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:28.028438 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:28.028237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:28.028438 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:28.028360 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:28.028438 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:28.028416 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs podName:8d832341-de73-4a7b-9ce0-c41bbe655035 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:30.028398149 +0000 UTC m=+5.274375747 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs") pod "network-metrics-daemon-cd7lc" (UID: "8d832341-de73-4a7b-9ce0-c41bbe655035") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:28.028683 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:28.028474 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:28.028683 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:28.028504 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret podName:8d5f4689-033d-4aa8-8603-a56cbdd449cb nodeName:}" failed. No retries permitted until 2026-11-03 06:58:30.028493955 +0000 UTC m=+5.274471550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret") pod "global-pull-secret-syncer-zhpcz" (UID: "8d5f4689-033d-4aa8-8603-a56cbdd449cb") : object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:28.128744 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:28.128688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9tc\" (UniqueName: \"kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc\") pod \"network-check-target-9gjvc\" (UID: \"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3\") " pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:28.129265 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:28.128888 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 06:58:28.129265 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:28.128916 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 06:58:28.129265 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:28.128931 2575 projected.go:196] Error preparing data for projected volume kube-api-access-fq9tc for pod openshift-network-diagnostics/network-check-target-9gjvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:28.129265 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:28.128994 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc podName:f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:30.12897584 +0000 UTC m=+5.374953436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq9tc" (UniqueName: "kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc") pod "network-check-target-9gjvc" (UID: "f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:28.223122 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:28.223080 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:28.223297 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:28.223221 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:28.223297 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:28.223293 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:28.223428 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:28.223365 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:28.223521 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:28.223432 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:28.223521 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:28.223498 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:28.473022 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:28.472877 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 06:53:26 +0000 UTC" deadline="2028-03-29 22:02:13.295386478 +0000 UTC" Nov 03 06:58:28.473022 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:28.472940 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12303h3m44.822450537s" Nov 03 06:58:30.046390 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:30.046357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:30.046844 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:30.046420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:30.046844 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:30.046552 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:30.046844 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:30.046614 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret podName:8d5f4689-033d-4aa8-8603-a56cbdd449cb nodeName:}" failed. No retries permitted until 2026-11-03 06:58:34.046596762 +0000 UTC m=+9.292574360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret") pod "global-pull-secret-syncer-zhpcz" (UID: "8d5f4689-033d-4aa8-8603-a56cbdd449cb") : object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:30.046844 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:30.046675 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:30.046844 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:30.046727 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs podName:8d832341-de73-4a7b-9ce0-c41bbe655035 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:34.046696229 +0000 UTC m=+9.292673811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs") pod "network-metrics-daemon-cd7lc" (UID: "8d832341-de73-4a7b-9ce0-c41bbe655035") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:30.147779 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:30.147443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9tc\" (UniqueName: \"kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc\") pod \"network-check-target-9gjvc\" (UID: \"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3\") " pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:30.147779 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:30.147598 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 06:58:30.147779 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:30.147613 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 06:58:30.147779 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:30.147624 2575 projected.go:196] Error preparing data for projected volume kube-api-access-fq9tc for pod openshift-network-diagnostics/network-check-target-9gjvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:30.147779 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:30.147680 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc podName:f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:34.147660365 +0000 UTC m=+9.393637961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq9tc" (UniqueName: "kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc") pod "network-check-target-9gjvc" (UID: "f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:30.223742 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:30.223629 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:30.223909 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:30.223796 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:30.223909 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:30.223860 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:30.224013 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:30.223931 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:30.224013 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:30.223969 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:30.224119 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:30.224029 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:31.273513 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:31.273288 2575 generic.go:356] "Generic (PLEG): container finished" podID="3e9850e7-1c78-455a-b2d0-77971e830f80" containerID="e37c2cafcd5016a24b8b0f7427204d32e7c538787b06711376b69359457e9650" exitCode=0 Nov 03 06:58:31.273973 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:31.273535 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ndld" event={"ID":"3e9850e7-1c78-455a-b2d0-77971e830f80","Type":"ContainerDied","Data":"e37c2cafcd5016a24b8b0f7427204d32e7c538787b06711376b69359457e9650"} Nov 03 06:58:32.223479 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:32.223411 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:32.223651 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:32.223562 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:32.223733 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:32.223662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:32.223833 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:32.223777 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:32.223918 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:32.223867 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:32.224011 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:32.223967 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:32.281118 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:32.281081 2575 generic.go:356] "Generic (PLEG): container finished" podID="c99460852de5d8046fca947f507d3971" containerID="dfd07c97b82fef8c06e475b8ca472ee9f567ad3228198a1ee86d88f5b5165d8e" exitCode=0 Nov 03 06:58:32.281519 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:32.281166 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" event={"ID":"c99460852de5d8046fca947f507d3971","Type":"ContainerDied","Data":"dfd07c97b82fef8c06e475b8ca472ee9f567ad3228198a1ee86d88f5b5165d8e"} Nov 03 06:58:32.284391 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:32.284363 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" event={"ID":"305af933eb9c6382ff5eabda70a436a6","Type":"ContainerStarted","Data":"8bb31c6c3ec9aec7b2ef09d1261e7cd4d4134b13e98605ac4c6e6848e3d574f0"} Nov 03 06:58:33.290912 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:33.290865 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" event={"ID":"fe326448-785b-4e7a-a189-76298b83c9b3","Type":"ContainerStarted","Data":"e0a69c7c4cb4766527e98b5955df781c8b44472e32288f96cd3e4cf305159a30"} Nov 03 06:58:33.293501 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:33.293472 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" event={"ID":"c99460852de5d8046fca947f507d3971","Type":"ContainerStarted","Data":"f6e5cb05cd3fa4c6752b527eaaf12dff7095b9c4642c4ba58f8ced498edcc9cf"} Nov 03 06:58:33.297624 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:33.297559 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nn9v4" event={"ID":"9fbdc7ee-c57c-4a65-ba85-b20706762749","Type":"ContainerStarted","Data":"8c429c72635f24a203a0e912110384024a8af390450b82053f8f4b3a73264a81"} Nov 03 06:58:33.414092 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:33.414030 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" podStartSLOduration=8.414012429 podStartE2EDuration="8.414012429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:32.522603969 +0000 UTC m=+7.768581572" watchObservedRunningTime="2026-11-03 06:58:33.414012429 +0000 UTC m=+8.659990037" Nov 03 06:58:33.543899 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:33.543780 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" podStartSLOduration=8.543761266 podStartE2EDuration="8.543761266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:33.414900762 +0000 UTC m=+8.660878368" watchObservedRunningTime="2026-11-03 06:58:33.543761266 +0000 UTC m=+8.789738870" Nov 03 06:58:34.084130 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:34.084096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:34.084299 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:34.084165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:34.084358 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:34.084300 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:34.084412 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:34.084356 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs podName:8d832341-de73-4a7b-9ce0-c41bbe655035 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:42.084339029 +0000 UTC m=+17.330316628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs") pod "network-metrics-daemon-cd7lc" (UID: "8d832341-de73-4a7b-9ce0-c41bbe655035") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:34.084476 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:34.084422 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:34.084476 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:34.084465 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret podName:8d5f4689-033d-4aa8-8603-a56cbdd449cb nodeName:}" failed. No retries permitted until 2026-11-03 06:58:42.084453242 +0000 UTC m=+17.330430830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret") pod "global-pull-secret-syncer-zhpcz" (UID: "8d5f4689-033d-4aa8-8603-a56cbdd449cb") : object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:34.185568 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:34.185536 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9tc\" (UniqueName: \"kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc\") pod \"network-check-target-9gjvc\" (UID: \"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3\") " pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:34.185758 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:34.185728 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 06:58:34.185758 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:34.185748 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 06:58:34.185758 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:34.185759 2575 projected.go:196] Error preparing data for projected volume kube-api-access-fq9tc for pod openshift-network-diagnostics/network-check-target-9gjvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:34.185947 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:34.185817 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc podName:f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:42.185799707 +0000 UTC m=+17.431777297 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq9tc" (UniqueName: "kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc") pod "network-check-target-9gjvc" (UID: "f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:34.224014 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:34.223563 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:34.224014 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:34.223702 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:34.224014 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:34.223775 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:34.224014 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:34.223841 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:34.224014 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:34.223893 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:34.224014 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:34.223975 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:34.918265 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:34.918234 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 06:58:35.179432 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:35.178791 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T06:58:34.918261902Z","UUID":"fa943d8e-2a04-4464-bf55-fb1a45622211","Handler":null,"Name":"","Endpoint":""} Nov 03 06:58:35.182352 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:35.182327 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 06:58:35.182500 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:35.182362 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 06:58:35.306417 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:35.306380 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rww5p" event={"ID":"6551e74b-f4c1-4be3-8fbf-cf5f262caea4","Type":"ContainerStarted","Data":"ac7d362b09eedb7c34e2b07b20f9141220317993e392c99f04c9ea44ef7894aa"} Nov 03 06:58:35.309782 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:35.309704 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" event={"ID":"fe326448-785b-4e7a-a189-76298b83c9b3","Type":"ContainerStarted","Data":"541efa8001f6ee8d24e9e1036d13a6062f4246d2a057c2487d790b4a9e96cb38"} Nov 03 06:58:35.428405 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:35.428354 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nn9v4" podStartSLOduration=4.804011972 podStartE2EDuration="10.428336925s" totalImagesPullingTime="5.624324953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:33.544508664 +0000 UTC m=+8.790486267" watchObservedRunningTime="2026-11-03 06:58:35.428336925 +0000 UTC m=+10.674314529" Nov 03 06:58:36.223944 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:36.223911 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:36.223944 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:36.223935 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:36.223944 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:36.223952 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:36.224464 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:36.224055 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:36.224464 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:36.224129 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:36.224464 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:36.224195 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:36.313675 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:36.313638 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mwws4" event={"ID":"2fa0cc98-1f99-4197-9a87-52c9ea4277f9","Type":"ContainerStarted","Data":"9bf35f133cf329ee127f0540ad625ada053307522e1aa30b6c8e3f06f46bfaf5"} Nov 03 06:58:36.316823 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:36.316795 2575 generic.go:356] "Generic (PLEG): container finished" podID="3e9850e7-1c78-455a-b2d0-77971e830f80" containerID="a1968da4a34f6ecbf768237625ace6f404ce71f008e0e2b72970a8d59dcc9a54" exitCode=0 Nov 03 06:58:36.316939 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:36.316823 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ndld" event={"ID":"3e9850e7-1c78-455a-b2d0-77971e830f80","Type":"ContainerDied","Data":"a1968da4a34f6ecbf768237625ace6f404ce71f008e0e2b72970a8d59dcc9a54"} Nov 03 06:58:36.439055 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:36.438844 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-mwws4" podStartSLOduration=2.8241934239999997 podStartE2EDuration="11.438827722s" totalImagesPullingTime="8.614634298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:36.438534166 +0000 UTC m=+11.684511771" watchObservedRunningTime="2026-11-03 06:58:36.438827722 +0000 UTC m=+11.684805327" Nov 03 06:58:36.439413 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:36.439381 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rww5p" podStartSLOduration=3.2234060429999998 podStartE2EDuration="11.439367411s" totalImagesPullingTime="8.215961368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:35.429379829 +0000 UTC m=+10.675357433" watchObservedRunningTime="2026-11-03 06:58:36.439367411 +0000 UTC m=+11.685345018" Nov 03 06:58:37.322284 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:37.322243 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" event={"ID":"fe326448-785b-4e7a-a189-76298b83c9b3","Type":"ContainerStarted","Data":"64bb1c8737bdfb205420f83308133894db960b292941740f3e78b594b3971a48"} Nov 03 06:58:37.326938 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:37.326901 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s5f7z" event={"ID":"eaee23aa-3918-44a3-a35c-7a77133e45c7","Type":"ContainerStarted","Data":"8db4a395ec15585d773dc2ce02b0d0e0214238273c9fb0a383a30fd27a0d34dd"} Nov 03 06:58:37.446083 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:37.446032 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" podStartSLOduration=3.175737133 podStartE2EDuration="12.44601464s" totalImagesPullingTime="9.270277507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:37.445251343 +0000 UTC m=+12.691228946" watchObservedRunningTime="2026-11-03 06:58:37.44601464 +0000 UTC m=+12.691992243" Nov 03 06:58:37.564544 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:37.564484 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s5f7z" podStartSLOduration=3.719117487 podStartE2EDuration="12.564463577s" totalImagesPullingTime="8.84534609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:37.564429884 +0000 UTC m=+12.810407489" watchObservedRunningTime="2026-11-03 06:58:37.564463577 +0000 UTC m=+12.810441184" Nov 03 06:58:38.223164 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:38.223126 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:38.223358 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:38.223255 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:38.223358 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:38.223321 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:38.223445 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:38.223383 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:38.223668 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:38.223643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:38.223814 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:38.223783 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:38.330251 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:38.330208 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6f246" event={"ID":"46730579-83ec-4a31-8a10-759a76d63eac","Type":"ContainerStarted","Data":"896c4bd848930b5ac93b89c4155638e5955cc61a5d0f9d59799b1aecd67040e4"} Nov 03 06:58:38.333040 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:38.332998 2575 generic.go:356] "Generic (PLEG): container finished" podID="3e9850e7-1c78-455a-b2d0-77971e830f80" containerID="9ba7809bc91c7ea0179d7eabdc4b555a04fb9746fc3f77ba198c19366fd2fcd1" exitCode=0 Nov 03 06:58:38.333169 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:38.333065 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ndld" event={"ID":"3e9850e7-1c78-455a-b2d0-77971e830f80","Type":"ContainerDied","Data":"9ba7809bc91c7ea0179d7eabdc4b555a04fb9746fc3f77ba198c19366fd2fcd1"} Nov 03 06:58:38.593124 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:38.593057 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6f246" podStartSLOduration=3.048192102 podStartE2EDuration="13.593036806s" totalImagesPullingTime="10.544844704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:38.592485544 +0000 UTC m=+13.838463150" watchObservedRunningTime="2026-11-03 06:58:38.593036806 +0000 UTC m=+13.839014409" Nov 03 06:58:40.223294 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:40.223244 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:40.223294 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:40.223295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:40.224110 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:40.223260 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:40.224110 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:40.223391 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:40.224110 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:40.223471 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:40.224110 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:40.223557 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:40.340931 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:40.340896 2575 generic.go:356] "Generic (PLEG): container finished" podID="3e9850e7-1c78-455a-b2d0-77971e830f80" containerID="902485287b93715ebabc966956bb67a886ff1a6519b161e106a76d6031cd98a7" exitCode=0 Nov 03 06:58:40.341073 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:40.340956 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ndld" event={"ID":"3e9850e7-1c78-455a-b2d0-77971e830f80","Type":"ContainerDied","Data":"902485287b93715ebabc966956bb67a886ff1a6519b161e106a76d6031cd98a7"} Nov 03 06:58:41.345324 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:41.345129 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nj9qb" event={"ID":"eff68601-5319-4dc8-93c6-16de197bd812","Type":"ContainerStarted","Data":"482defbf63a2835608c6a40cee418fc4815bf2696b426e659bf1d646325d3809"} Nov 03 06:58:41.464567 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:41.464508 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nj9qb" podStartSLOduration=2.748767125 podStartE2EDuration="16.464491158s" totalImagesPullingTime="13.715724033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-11-03 06:58:41.464354034 +0000 UTC m=+16.710331638" watchObservedRunningTime="2026-11-03 06:58:41.464491158 +0000 UTC m=+16.710468763" Nov 03 06:58:42.149123 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.149089 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:42.149275 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.149172 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:42.149275 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:42.149236 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:42.149380 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:42.149285 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:42.149380 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:42.149306 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs podName:8d832341-de73-4a7b-9ce0-c41bbe655035 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:58.149285675 +0000 UTC m=+33.395263269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs") pod "network-metrics-daemon-cd7lc" (UID: "8d832341-de73-4a7b-9ce0-c41bbe655035") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 06:58:42.149380 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:42.149334 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret podName:8d5f4689-033d-4aa8-8603-a56cbdd449cb nodeName:}" failed. No retries permitted until 2026-11-03 06:58:58.149319659 +0000 UTC m=+33.395297246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret") pod "global-pull-secret-syncer-zhpcz" (UID: "8d5f4689-033d-4aa8-8603-a56cbdd449cb") : object "kube-system"/"original-pull-secret" not registered Nov 03 06:58:42.223556 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.223524 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:42.223661 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.223630 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:42.223797 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.223778 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:42.223857 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:42.223792 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:42.223925 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:42.223902 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:42.224014 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:42.223997 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:42.250007 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.249976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9tc\" (UniqueName: \"kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc\") pod \"network-check-target-9gjvc\" (UID: \"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3\") " pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:42.250202 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:42.250180 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 06:58:42.250255 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:42.250212 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 06:58:42.250255 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:42.250224 2575 projected.go:196] Error preparing data for projected volume kube-api-access-fq9tc for pod openshift-network-diagnostics/network-check-target-9gjvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:42.250345 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:42.250285 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc podName:f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:58.250266789 +0000 UTC m=+33.496244375 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fq9tc" (UniqueName: "kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc") pod "network-check-target-9gjvc" (UID: "f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 06:58:42.351773 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.351534 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 06:58:42.352268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.352163 2575 generic.go:356] "Generic (PLEG): container finished" podID="3378427f-b215-4a67-8a9d-aab1f8b05d8f" containerID="620610d68e5af2f05411b0637db646d58e3676d95339a809e80dd89d665a2a99" exitCode=1 Nov 03 06:58:42.352268 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.352227 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" event={"ID":"3378427f-b215-4a67-8a9d-aab1f8b05d8f","Type":"ContainerStarted","Data":"b7c87720084bf1a4e8f72d104f338e4f4e4711c8f3109b580f10f508d598cb7b"} Nov 03 06:58:42.352358 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.352307 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" event={"ID":"3378427f-b215-4a67-8a9d-aab1f8b05d8f","Type":"ContainerStarted","Data":"b6f06af87eaa3a29d6c79c11168201d12c4defab012d6916cb80404d17c6f617"} Nov 03 06:58:42.352409 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.352361 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" event={"ID":"3378427f-b215-4a67-8a9d-aab1f8b05d8f","Type":"ContainerStarted","Data":"5a06783e476ac7477ead48d7564366c8663b42079421ffb08549044f5ef6c507"} Nov 03 06:58:42.352409 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.352391 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" event={"ID":"3378427f-b215-4a67-8a9d-aab1f8b05d8f","Type":"ContainerStarted","Data":"c78cfa26c7cbe43d176cfa453daae95a863a462865b2ca4a3769bda286ff278a"} Nov 03 06:58:42.352476 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.352422 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" event={"ID":"3378427f-b215-4a67-8a9d-aab1f8b05d8f","Type":"ContainerDied","Data":"620610d68e5af2f05411b0637db646d58e3676d95339a809e80dd89d665a2a99"} Nov 03 06:58:42.352476 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:42.352469 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" event={"ID":"3378427f-b215-4a67-8a9d-aab1f8b05d8f","Type":"ContainerStarted","Data":"8641e3dc5b08b98f60f43c39d975c66958ce71e1eda36ad9f3e69f7b70d9548b"} Nov 03 06:58:43.356479 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:43.356442 2575 generic.go:356] "Generic (PLEG): container finished" podID="3e9850e7-1c78-455a-b2d0-77971e830f80" containerID="a293e9b889349b49c758759a5735efb18b264439134b8affbaa7762b86090bc9" exitCode=0 Nov 03 06:58:43.357000 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:43.356510 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ndld" event={"ID":"3e9850e7-1c78-455a-b2d0-77971e830f80","Type":"ContainerDied","Data":"a293e9b889349b49c758759a5735efb18b264439134b8affbaa7762b86090bc9"} Nov 03 06:58:44.223559 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:44.223518 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:44.223559 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:44.223554 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:44.223810 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:44.223563 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:44.223810 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:44.223640 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:44.223810 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:44.223690 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:44.223810 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:44.223784 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:44.361324 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:44.361293 2575 generic.go:356] "Generic (PLEG): container finished" podID="3e9850e7-1c78-455a-b2d0-77971e830f80" containerID="8259cceb94ed89a2d67fc53a8b24fb88493d6932efc57596689d941f15d2b3c3" exitCode=0 Nov 03 06:58:44.361699 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:44.361349 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ndld" event={"ID":"3e9850e7-1c78-455a-b2d0-77971e830f80","Type":"ContainerDied","Data":"8259cceb94ed89a2d67fc53a8b24fb88493d6932efc57596689d941f15d2b3c3"} Nov 03 06:58:44.436394 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:44.436365 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:58:44.436907 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:44.436887 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:58:45.365974 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:45.365950 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 06:58:45.366512 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:45.366344 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" event={"ID":"3378427f-b215-4a67-8a9d-aab1f8b05d8f","Type":"ContainerStarted","Data":"d72781d6fdc316b6607c1f7a6afbee8f404f3820d0f5399a38010999dd6f3263"} Nov 03 06:58:45.369704 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:45.369678 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ndld" event={"ID":"3e9850e7-1c78-455a-b2d0-77971e830f80","Type":"ContainerStarted","Data":"70bc79fbc6fbfb2c8fbd6c7de7350ad8a282144177b361b442a9cffb4111d46f"} Nov 03 06:58:45.370031 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:45.370011 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:58:45.370272 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:45.370257 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:58:45.496733 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:45.496671 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4ndld" podStartSLOduration=8.526137838 podStartE2EDuration="20.49665841s" totalImagesPullingTime="11.970520572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:45.495998825 +0000 UTC m=+20.741976429" watchObservedRunningTime="2026-11-03 06:58:45.49665841 +0000 UTC m=+20.742636014" Nov 03 06:58:46.222992 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:46.222965 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:46.223199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:46.222972 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:46.223199 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:46.223073 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:46.223199 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:46.223092 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:46.223199 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:46.223154 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:46.223403 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:46.223237 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:47.377516 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:47.377353 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 06:58:47.378197 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:47.377917 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" event={"ID":"3378427f-b215-4a67-8a9d-aab1f8b05d8f","Type":"ContainerStarted","Data":"9ced1200241075fbb6e13d06e5ff8df9e3064127c45303cdc62ed2d63704cd23"} Nov 03 06:58:47.378298 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:47.378281 2575 scope.go:122] "RemoveContainer" containerID="620610d68e5af2f05411b0637db646d58e3676d95339a809e80dd89d665a2a99" Nov 03 06:58:48.223851 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:48.223816 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:48.223851 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:48.223843 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:48.224119 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:48.223855 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:48.224119 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:48.223934 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:48.224119 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:48.223970 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:48.224119 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:48.224052 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:48.383048 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:48.383028 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 06:58:48.383434 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:48.383366 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" event={"ID":"3378427f-b215-4a67-8a9d-aab1f8b05d8f","Type":"ContainerStarted","Data":"1fb4f029e8818487fdeb18e0b78920f3e98a21806f6b57b96a0ffce5dd421e6b"} Nov 03 06:58:48.383474 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:48.383453 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 06:58:48.383627 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:48.383612 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:48.383704 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:48.383692 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:48.398692 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:48.398662 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:48.398866 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:48.398849 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:48.619190 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:48.619149 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" podStartSLOduration=8.56737712 podStartE2EDuration="23.619136386s" totalImagesPullingTime="15.051759266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:58:48.568171872 +0000 UTC m=+23.814149475" watchObservedRunningTime="2026-11-03 06:58:48.619136386 +0000 UTC m=+23.865113990" Nov 03 06:58:49.194129 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:49.194102 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zhpcz"] Nov 03 06:58:49.194292 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:49.194196 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:49.194292 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:49.194275 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:49.197232 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:49.197207 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9gjvc"] Nov 03 06:58:49.197350 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:49.197294 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:49.197509 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:49.197480 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:49.197845 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:49.197825 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cd7lc"] Nov 03 06:58:49.197924 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:49.197907 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:49.198018 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:49.198001 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:49.385689 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:49.385663 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 06:58:49.966428 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:49.966398 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:58:51.226532 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:51.226505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:51.226864 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:51.226539 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:51.226864 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:51.226726 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:51.226864 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:51.226697 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:51.226864 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:51.226799 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:51.227056 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:51.226862 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:51.401707 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:51.401662 2575 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" podUID="3378427f-b215-4a67-8a9d-aab1f8b05d8f" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 06:58:53.223851 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:53.223820 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:53.224206 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:53.223861 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:53.224206 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:53.223818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:53.224206 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:53.223970 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:53.224206 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:53.224010 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:53.224206 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:53.224062 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:55.224385 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:55.224168 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:55.224812 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:55.224210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:55.224812 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:55.224478 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gjvc" podUID="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" Nov 03 06:58:55.224812 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:55.224235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:55.224812 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:55.224537 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd7lc" podUID="8d832341-de73-4a7b-9ce0-c41bbe655035" Nov 03 06:58:55.224812 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:55.224609 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhpcz" podUID="8d5f4689-033d-4aa8-8603-a56cbdd449cb" Nov 03 06:58:56.029647 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.029578 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeReady" Nov 03 06:58:56.029821 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.029672 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 06:58:56.147826 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.147799 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kkrxc"] Nov 03 06:58:56.206815 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.206786 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-htqcm"] Nov 03 06:58:56.206949 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.206896 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kkrxc" Nov 03 06:58:56.222070 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.222003 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kkrxc"] Nov 03 06:58:56.222174 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.222081 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.222174 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.222101 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-htqcm"] Nov 03 06:58:56.264407 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.264384 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 06:58:56.264761 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.264537 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 06:58:56.264761 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.264583 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-kl8t2\"" Nov 03 06:58:56.264761 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.264633 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 06:58:56.276265 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.276252 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 06:58:56.276341 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.276302 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 06:58:56.276396 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.276385 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nd4b5\"" Nov 03 06:58:56.353240 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.353221 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmkk6\" (UniqueName: \"kubernetes.io/projected/1a76e41e-99b4-4768-a589-014a65e868ac-kube-api-access-lmkk6\") pod \"ingress-canary-kkrxc\" (UID: \"1a76e41e-99b4-4768-a589-014a65e868ac\") " pod="openshift-ingress-canary/ingress-canary-kkrxc" Nov 03 06:58:56.353342 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.353263 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a76e41e-99b4-4768-a589-014a65e868ac-cert\") pod \"ingress-canary-kkrxc\" (UID: \"1a76e41e-99b4-4768-a589-014a65e868ac\") " pod="openshift-ingress-canary/ingress-canary-kkrxc" Nov 03 06:58:56.353342 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.353283 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.353409 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.353344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jtj6\" (UniqueName: \"kubernetes.io/projected/d399cdee-b235-47b8-96e5-1687506510d0-kube-api-access-4jtj6\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.353409 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.353376 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d399cdee-b235-47b8-96e5-1687506510d0-tmp-dir\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.353467 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.353450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d399cdee-b235-47b8-96e5-1687506510d0-config-volume\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.454374 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.454351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d399cdee-b235-47b8-96e5-1687506510d0-config-volume\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.454484 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.454384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmkk6\" (UniqueName: \"kubernetes.io/projected/1a76e41e-99b4-4768-a589-014a65e868ac-kube-api-access-lmkk6\") pod \"ingress-canary-kkrxc\" (UID: \"1a76e41e-99b4-4768-a589-014a65e868ac\") " pod="openshift-ingress-canary/ingress-canary-kkrxc" Nov 03 06:58:56.454484 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.454412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a76e41e-99b4-4768-a589-014a65e868ac-cert\") pod \"ingress-canary-kkrxc\" (UID: \"1a76e41e-99b4-4768-a589-014a65e868ac\") " pod="openshift-ingress-canary/ingress-canary-kkrxc" Nov 03 06:58:56.454484 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.454428 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.454621 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:56.454500 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:58:56.454621 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:56.454542 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls podName:d399cdee-b235-47b8-96e5-1687506510d0 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:56.954528293 +0000 UTC m=+32.200505874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls") pod "dns-default-htqcm" (UID: "d399cdee-b235-47b8-96e5-1687506510d0") : secret "dns-default-metrics-tls" not found Nov 03 06:58:56.454621 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.454555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jtj6\" (UniqueName: \"kubernetes.io/projected/d399cdee-b235-47b8-96e5-1687506510d0-kube-api-access-4jtj6\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.454621 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.454578 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d399cdee-b235-47b8-96e5-1687506510d0-tmp-dir\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.454844 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.454834 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d399cdee-b235-47b8-96e5-1687506510d0-tmp-dir\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.455055 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.455030 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d399cdee-b235-47b8-96e5-1687506510d0-config-volume\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.458472 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.458453 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a76e41e-99b4-4768-a589-014a65e868ac-cert\") pod \"ingress-canary-kkrxc\" (UID: \"1a76e41e-99b4-4768-a589-014a65e868ac\") " pod="openshift-ingress-canary/ingress-canary-kkrxc" Nov 03 06:58:56.518697 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.518674 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jtj6\" (UniqueName: \"kubernetes.io/projected/d399cdee-b235-47b8-96e5-1687506510d0-kube-api-access-4jtj6\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.518783 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.518749 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmkk6\" (UniqueName: \"kubernetes.io/projected/1a76e41e-99b4-4768-a589-014a65e868ac-kube-api-access-lmkk6\") pod \"ingress-canary-kkrxc\" (UID: \"1a76e41e-99b4-4768-a589-014a65e868ac\") " pod="openshift-ingress-canary/ingress-canary-kkrxc" Nov 03 06:58:56.529581 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.529565 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kkrxc" Nov 03 06:58:56.957697 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.957633 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:56.957844 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:56.957747 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:58:56.957844 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:56.957805 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls podName:d399cdee-b235-47b8-96e5-1687506510d0 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:57.95779326 +0000 UTC m=+33.203770841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls") pod "dns-default-htqcm" (UID: "d399cdee-b235-47b8-96e5-1687506510d0") : secret "dns-default-metrics-tls" not found Nov 03 06:58:56.993473 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:56.993449 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kkrxc"] Nov 03 06:58:56.996302 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:56.996271 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a76e41e_99b4_4768_a589_014a65e868ac.slice/crio-f1e02fdc9a47d6890e3bc2b4dcd98300605fabf0256f70ddc018054ee9aa8c25 WatchSource:0}: Error finding container f1e02fdc9a47d6890e3bc2b4dcd98300605fabf0256f70ddc018054ee9aa8c25: Status 404 returned error can't find the container with id f1e02fdc9a47d6890e3bc2b4dcd98300605fabf0256f70ddc018054ee9aa8c25 Nov 03 06:58:57.223537 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.223473 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:57.223634 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.223479 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:57.223683 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.223645 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:57.280819 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.280789 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 06:58:57.281152 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.280856 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kx4sk\"" Nov 03 06:58:57.281152 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.280858 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dftjg\"" Nov 03 06:58:57.281152 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.280937 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tq2gc\"" Nov 03 06:58:57.281152 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.280985 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 06:58:57.281152 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.280989 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 06:58:57.281152 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.281019 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 06:58:57.300824 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.300799 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw"] Nov 03 06:58:57.335532 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.335510 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw" Nov 03 06:58:57.361972 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.361947 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw"] Nov 03 06:58:57.391673 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.391651 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wqbfp"] Nov 03 06:58:57.399414 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.399396 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 06:58:57.399515 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.399470 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 06:58:57.399663 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.399649 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-p8ttv\"" Nov 03 06:58:57.402871 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.402858 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.409014 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.408982 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kkrxc" event={"ID":"1a76e41e-99b4-4768-a589-014a65e868ac","Type":"ContainerStarted","Data":"f1e02fdc9a47d6890e3bc2b4dcd98300605fabf0256f70ddc018054ee9aa8c25"} Nov 03 06:58:57.456086 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.456060 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqbfp"] Nov 03 06:58:57.460539 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.460517 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2bnh\" (UniqueName: \"kubernetes.io/projected/7627e405-9b18-41b2-9fdb-e39eb1ef7e3b-kube-api-access-b2bnh\") pod \"migrator-66df5d496b-2tcnw\" (UID: \"7627e405-9b18-41b2-9fdb-e39eb1ef7e3b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw" Nov 03 06:58:57.499647 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.499581 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 06:58:57.499647 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.499593 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 06:58:57.499647 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.499607 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 06:58:57.499647 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.499627 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 06:58:57.499857 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.499804 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7dznd\"" Nov 03 06:58:57.561539 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.561514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bnh\" (UniqueName: \"kubernetes.io/projected/7627e405-9b18-41b2-9fdb-e39eb1ef7e3b-kube-api-access-b2bnh\") pod \"migrator-66df5d496b-2tcnw\" (UID: \"7627e405-9b18-41b2-9fdb-e39eb1ef7e3b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw" Nov 03 06:58:57.561647 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.561544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.561647 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.561574 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0fc916e0-504f-4aea-85ac-37562156d41e-crio-socket\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.561647 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.561589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjn2\" (UniqueName: \"kubernetes.io/projected/0fc916e0-504f-4aea-85ac-37562156d41e-kube-api-access-9wjn2\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.561833 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.561660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.561833 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.561761 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fc916e0-504f-4aea-85ac-37562156d41e-data-volume\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.629677 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.629644 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bnh\" (UniqueName: \"kubernetes.io/projected/7627e405-9b18-41b2-9fdb-e39eb1ef7e3b-kube-api-access-b2bnh\") pod \"migrator-66df5d496b-2tcnw\" (UID: \"7627e405-9b18-41b2-9fdb-e39eb1ef7e3b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw" Nov 03 06:58:57.662643 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.662615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.662782 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.662660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0fc916e0-504f-4aea-85ac-37562156d41e-crio-socket\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.662782 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.662682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjn2\" (UniqueName: \"kubernetes.io/projected/0fc916e0-504f-4aea-85ac-37562156d41e-kube-api-access-9wjn2\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.662782 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.662742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.662941 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:57.662779 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:58:57.662941 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.662790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fc916e0-504f-4aea-85ac-37562156d41e-data-volume\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.662941 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.662845 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0fc916e0-504f-4aea-85ac-37562156d41e-crio-socket\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.662941 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:57.662853 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls podName:0fc916e0-504f-4aea-85ac-37562156d41e nodeName:}" failed. No retries permitted until 2026-11-03 06:58:58.16283525 +0000 UTC m=+33.408812846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqbfp" (UID: "0fc916e0-504f-4aea-85ac-37562156d41e") : secret "insights-runtime-extractor-tls" not found Nov 03 06:58:57.662941 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:57.662875 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 06:58:57.663125 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:57.662961 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm podName:0fc916e0-504f-4aea-85ac-37562156d41e nodeName:}" failed. No retries permitted until 2026-11-03 06:58:58.162944008 +0000 UTC m=+33.408921612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqbfp" (UID: "0fc916e0-504f-4aea-85ac-37562156d41e") : configmap "kube-rbac-proxy" not found Nov 03 06:58:57.663170 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.663152 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fc916e0-504f-4aea-85ac-37562156d41e-data-volume\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.666644 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.666629 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw" Nov 03 06:58:57.734891 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.734851 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjn2\" (UniqueName: \"kubernetes.io/projected/0fc916e0-504f-4aea-85ac-37562156d41e-kube-api-access-9wjn2\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:57.964944 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:57.964908 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:57.965211 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:57.965081 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:58:57.965211 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:57.965182 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls podName:d399cdee-b235-47b8-96e5-1687506510d0 nodeName:}" failed. No retries permitted until 2026-11-03 06:58:59.965160057 +0000 UTC m=+35.211137667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls") pod "dns-default-htqcm" (UID: "d399cdee-b235-47b8-96e5-1687506510d0") : secret "dns-default-metrics-tls" not found Nov 03 06:58:58.054691 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.054469 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw"] Nov 03 06:58:58.057253 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:58.057225 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7627e405_9b18_41b2_9fdb_e39eb1ef7e3b.slice/crio-022a29ea035b9e67d182c0eae1955a7cfb3fab7a76d69e2de8d531882bf937be WatchSource:0}: Error finding container 022a29ea035b9e67d182c0eae1955a7cfb3fab7a76d69e2de8d531882bf937be: Status 404 returned error can't find the container with id 022a29ea035b9e67d182c0eae1955a7cfb3fab7a76d69e2de8d531882bf937be Nov 03 06:58:58.165983 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.165953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:58.166144 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.165993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:58:58.166144 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:58.166082 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 06:58:58.166144 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:58.166098 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 06:58:58.166144 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.166134 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:58.166342 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:58.166149 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm podName:0fc916e0-504f-4aea-85ac-37562156d41e nodeName:}" failed. No retries permitted until 2026-11-03 06:58:59.16612776 +0000 UTC m=+34.412105348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqbfp" (UID: "0fc916e0-504f-4aea-85ac-37562156d41e") : configmap "kube-rbac-proxy" not found Nov 03 06:58:58.166342 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:58.166219 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs podName:8d832341-de73-4a7b-9ce0-c41bbe655035 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:30.16620352 +0000 UTC m=+65.412181102 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs") pod "network-metrics-daemon-cd7lc" (UID: "8d832341-de73-4a7b-9ce0-c41bbe655035") : secret "metrics-daemon-secret" not found Nov 03 06:58:58.166342 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.166249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:58.166554 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:58.166360 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:58:58.166554 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:58.166398 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls podName:0fc916e0-504f-4aea-85ac-37562156d41e nodeName:}" failed. No retries permitted until 2026-11-03 06:58:59.16638641 +0000 UTC m=+34.412363992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqbfp" (UID: "0fc916e0-504f-4aea-85ac-37562156d41e") : secret "insights-runtime-extractor-tls" not found Nov 03 06:58:58.168732 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.168682 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:58.266729 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.266654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9tc\" (UniqueName: \"kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc\") pod \"network-check-target-9gjvc\" (UID: \"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3\") " pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:58.269539 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.269518 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq9tc\" (UniqueName: \"kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc\") pod \"network-check-target-9gjvc\" (UID: \"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3\") " pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:58.412602 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.412564 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw" event={"ID":"7627e405-9b18-41b2-9fdb-e39eb1ef7e3b","Type":"ContainerStarted","Data":"022a29ea035b9e67d182c0eae1955a7cfb3fab7a76d69e2de8d531882bf937be"} Nov 03 06:58:58.453451 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.453419 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:58:58.469773 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.469751 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:58:58.878059 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.878035 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zhpcz"] Nov 03 06:58:58.880502 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:58.880477 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5f4689_033d_4aa8_8603_a56cbdd449cb.slice/crio-c319cf8cfa9607a360398f41ee314fcaa87942c7629a2d36f80c193ea818b0f4 WatchSource:0}: Error finding container c319cf8cfa9607a360398f41ee314fcaa87942c7629a2d36f80c193ea818b0f4: Status 404 returned error can't find the container with id c319cf8cfa9607a360398f41ee314fcaa87942c7629a2d36f80c193ea818b0f4 Nov 03 06:58:58.930172 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:58.930144 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9gjvc"] Nov 03 06:58:58.933342 ip-10-0-138-121 kubenswrapper[2575]: W1103 06:58:58.933318 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a741d6_8992_4f8f_8cd7_94ea7c3ccff3.slice/crio-65e91977d8053582f57e74240c8ea6188ccfac691087adf44dc125d9fcea3ef3 WatchSource:0}: Error finding container 65e91977d8053582f57e74240c8ea6188ccfac691087adf44dc125d9fcea3ef3: Status 404 returned error can't find the container with id 65e91977d8053582f57e74240c8ea6188ccfac691087adf44dc125d9fcea3ef3 Nov 03 06:58:59.175807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:59.174850 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:59.175807 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:59.174939 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:58:59.175807 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:59.175097 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:58:59.175807 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:59.175148 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls podName:0fc916e0-504f-4aea-85ac-37562156d41e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:01.175134728 +0000 UTC m=+36.421112324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqbfp" (UID: "0fc916e0-504f-4aea-85ac-37562156d41e") : secret "insights-runtime-extractor-tls" not found Nov 03 06:58:59.175807 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:59.175618 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 06:58:59.175807 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:59.175725 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm podName:0fc916e0-504f-4aea-85ac-37562156d41e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:01.175692022 +0000 UTC m=+36.421669618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqbfp" (UID: "0fc916e0-504f-4aea-85ac-37562156d41e") : configmap "kube-rbac-proxy" not found Nov 03 06:58:59.416229 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:59.416189 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9gjvc" event={"ID":"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3","Type":"ContainerStarted","Data":"65e91977d8053582f57e74240c8ea6188ccfac691087adf44dc125d9fcea3ef3"} Nov 03 06:58:59.417826 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:59.417800 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zhpcz" event={"ID":"8d5f4689-033d-4aa8-8603-a56cbdd449cb","Type":"ContainerStarted","Data":"c319cf8cfa9607a360398f41ee314fcaa87942c7629a2d36f80c193ea818b0f4"} Nov 03 06:58:59.982137 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:58:59.982105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:58:59.982306 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:59.982289 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:58:59.982385 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:58:59.982372 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls podName:d399cdee-b235-47b8-96e5-1687506510d0 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:03.98235377 +0000 UTC m=+39.228331372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls") pod "dns-default-htqcm" (UID: "d399cdee-b235-47b8-96e5-1687506510d0") : secret "dns-default-metrics-tls" not found Nov 03 06:59:01.193384 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:59:01.193343 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:01.193883 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:59:01.193444 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:01.193883 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:59:01.193477 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 06:59:01.193883 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:59:01.193558 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:59:01.193883 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:59:01.193562 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm podName:0fc916e0-504f-4aea-85ac-37562156d41e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.193537329 +0000 UTC m=+40.439514919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqbfp" (UID: "0fc916e0-504f-4aea-85ac-37562156d41e") : configmap "kube-rbac-proxy" not found Nov 03 06:59:01.193883 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:59:01.193622 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls podName:0fc916e0-504f-4aea-85ac-37562156d41e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:05.19360694 +0000 UTC m=+40.439584527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqbfp" (UID: "0fc916e0-504f-4aea-85ac-37562156d41e") : secret "insights-runtime-extractor-tls" not found Nov 03 06:59:01.425604 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:59:01.425559 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kkrxc" event={"ID":"1a76e41e-99b4-4768-a589-014a65e868ac","Type":"ContainerStarted","Data":"da2318ebaacdf54bdf985894b1ce6219cef1e23e41edfa475844f875edc1a02f"} Nov 03 06:59:01.427864 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:59:01.427825 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw" event={"ID":"7627e405-9b18-41b2-9fdb-e39eb1ef7e3b","Type":"ContainerStarted","Data":"04cdeae8cfb2525cb2b40ca3b422f4ae9045a69eb4144afd8a3ccc89bd8adc64"} Nov 03 06:59:01.553003 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:59:01.552950 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kkrxc" podStartSLOduration=2.222597796 podStartE2EDuration="5.552936555s" totalImagesPullingTime="3.330338759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:01.552428154 +0000 UTC m=+36.798405760" watchObservedRunningTime="2026-11-03 06:59:01.552936555 +0000 UTC m=+36.798914159" Nov 03 06:59:01.672049 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:59:01.671987 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw" podStartSLOduration=2.414980167 podStartE2EDuration="4.671968228s" totalImagesPullingTime="2.256988061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:01.671175717 +0000 UTC m=+36.917153322" watchObservedRunningTime="2026-11-03 06:59:01.671968228 +0000 UTC m=+36.917945834" Nov 03 06:59:03.434358 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:59:03.434323 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9gjvc" event={"ID":"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3","Type":"ContainerStarted","Data":"f1e0f4e37a7c87a481d0e14d2c4a2beac78a86a6d12801bccc8c6ba541966152"} Nov 03 06:59:03.434824 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:59:03.434399 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:59:03.557583 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:59:03.557522 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9gjvc" podStartSLOduration=34.630392514 podStartE2EDuration="38.557505126s" totalImagesPullingTime="3.927112612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:03.556537681 +0000 UTC m=+38.802515286" watchObservedRunningTime="2026-11-03 06:59:03.557505126 +0000 UTC m=+38.803482761" Nov 03 06:59:04.020285 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:59:04.020252 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:04.020443 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:59:04.020409 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:59:04.020490 ip-10-0-138-121 kubenswrapper[2575]: E1103 06:59:04.020479 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls podName:d399cdee-b235-47b8-96e5-1687506510d0 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:12.020463834 +0000 UTC m=+47.266441440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls") pod "dns-default-htqcm" (UID: "d399cdee-b235-47b8-96e5-1687506510d0") : secret "dns-default-metrics-tls" not found Nov 03 06:59:04.714633 ip-10-0-138-121 kubenswrapper[2575]: I1103 06:59:04.714551 2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 06:59:04.714730 ip-10-0-138-121 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 06:59:04.732134 ip-10-0-138-121 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 06:59:04.732296 ip-10-0-138-121 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 06:59:04.732347 ip-10-0-138-121 systemd[1]: kubelet.service: Consumed 2.297s CPU time. Nov 03 06:59:04.742509 ip-10-0-138-121 systemd[1]: Starting Kubernetes Kubelet... Nov 03 06:59:04.834359 ip-10-0-138-121 kubenswrapper[4409]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 06:59:04.834359 ip-10-0-138-121 kubenswrapper[4409]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 06:59:04.834359 ip-10-0-138-121 kubenswrapper[4409]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 06:59:04.834359 ip-10-0-138-121 kubenswrapper[4409]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 06:59:04.837441 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837366 4409 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:04.837441 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837439 4409 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:04.837441 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837447 4409 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837452 4409 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837456 4409 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837460 4409 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837464 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837467 4409 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837471 4409 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837476 4409 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837480 4409 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837484 4409 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837488 4409 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837492 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837496 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837500 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837504 4409 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837508 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837512 4409 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837515 4409 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837519 4409 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:04.837655 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837523 4409 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837527 4409 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837531 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837534 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837538 4409 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837542 4409 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837546 4409 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837550 4409 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837554 4409 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837558 4409 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837563 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837568 4409 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837572 4409 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837577 4409 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837580 4409 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837585 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837589 4409 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837594 4409 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837598 4409 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837602 4409 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:04.838483 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837606 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837610 4409 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837614 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837619 4409 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837623 4409 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837627 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837632 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837636 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837640 4409 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837646 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837650 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837654 4409 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837658 4409 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837662 4409 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837666 4409 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837670 4409 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837674 4409 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837678 4409 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837683 4409 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837687 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:04.839424 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837691 4409 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837695 4409 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837699 4409 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837703 4409 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837708 4409 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837732 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837736 4409 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837740 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837744 4409 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837748 4409 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837753 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837757 4409 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837761 4409 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837765 4409 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837770 4409 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837775 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837779 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837784 4409 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837794 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:04.840307 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837799 4409 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837803 4409 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837808 4409 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837814 4409 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837819 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837823 4409 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837828 4409 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837832 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837836 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837841 4409 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837845 4409 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837849 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837853 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837857 4409 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837861 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837865 4409 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837870 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837874 4409 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837878 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837883 4409 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:04.841184 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837887 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837891 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837900 4409 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837907 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837912 4409 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.837917 4409 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839186 4409 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839199 4409 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839205 4409 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839211 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839216 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839221 4409 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839226 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839230 4409 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839234 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839238 4409 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839244 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839248 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:04.842116 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839253 4409 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839256 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839261 4409 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839265 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839269 4409 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839273 4409 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839277 4409 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839281 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839286 4409 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839291 4409 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839295 4409 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839300 4409 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839305 4409 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839311 4409 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839315 4409 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839319 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839323 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839327 4409 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839331 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839336 4409 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:04.842960 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839340 4409 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839344 4409 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839349 4409 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839353 4409 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839357 4409 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839361 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839365 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839369 4409 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839373 4409 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839377 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839381 4409 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839387 4409 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839392 4409 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839396 4409 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839400 4409 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839405 4409 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839410 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839414 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839418 4409 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:04.843789 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839422 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839426 4409 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839430 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839434 4409 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839439 4409 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839442 4409 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839446 4409 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839450 4409 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839455 4409 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839459 4409 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839463 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839468 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839472 4409 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839476 4409 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839480 4409 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839484 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839488 4409 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839492 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839497 4409 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839501 4409 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:04.844269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839505 4409 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839509 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839513 4409 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839518 4409 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839522 4409 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839527 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839531 4409 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839535 4409 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839539 4409 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839543 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839547 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839552 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839556 4409 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839564 4409 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839568 4409 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839572 4409 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839576 4409 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839580 4409 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839584 4409 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839588 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:04.844829 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839592 4409 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839596 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839600 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839604 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839608 4409 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839612 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839617 4409 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839621 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839625 4409 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839629 4409 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839633 4409 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839637 4409 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839644 4409 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839652 4409 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.839657 4409 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840496 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840505 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840511 4409 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:04.845379 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840515 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840521 4409 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840527 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840532 4409 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840536 4409 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840541 4409 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840545 4409 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840552 4409 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840557 4409 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840561 4409 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840565 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840569 4409 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840574 4409 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840580 4409 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840585 4409 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840590 4409 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840594 4409 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840598 4409 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:04.845851 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840602 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840606 4409 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840611 4409 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840615 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840620 4409 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840624 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840628 4409 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840632 4409 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840637 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840641 4409 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840645 4409 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840649 4409 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840653 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840657 4409 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840662 4409 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840666 4409 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840670 4409 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840674 4409 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840678 4409 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840682 4409 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:04.846284 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840687 4409 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840693 4409 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840697 4409 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840701 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840705 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840728 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840733 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840737 4409 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840741 4409 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840745 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840752 4409 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840756 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840760 4409 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840764 4409 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840770 4409 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840774 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840778 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840783 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840787 4409 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840791 4409 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:04.846804 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840795 4409 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840799 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840803 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840807 4409 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840812 4409 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840816 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840819 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840825 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840829 4409 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840833 4409 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840837 4409 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840841 4409 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840845 4409 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840851 4409 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840855 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840860 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840864 4409 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840868 4409 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840872 4409 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840876 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:04.847283 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840880 4409 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840885 4409 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840890 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840894 4409 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840898 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840902 4409 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840907 4409 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840911 4409 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840916 4409 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840920 4409 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840924 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840928 4409 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840934 4409 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840938 4409 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840942 4409 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840947 4409 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840951 4409 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840955 4409 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840960 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:04.847779 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840964 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840969 4409 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840973 4409 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840977 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840981 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.840986 4409 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841119 4409 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841131 4409 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841140 4409 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841147 4409 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841154 4409 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841159 4409 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841165 4409 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841171 4409 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841176 4409 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841183 4409 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841188 4409 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841193 4409 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841198 4409 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841202 4409 flags.go:64] FLAG: --cgroup-root="" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841207 4409 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841211 4409 flags.go:64] FLAG: --client-ca-file="" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841216 4409 flags.go:64] FLAG: --cloud-provider="external" Nov 03 06:59:04.848226 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841220 4409 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841228 4409 flags.go:64] FLAG: --cluster-domain="" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841232 4409 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841237 4409 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841242 4409 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841247 4409 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841253 4409 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841258 4409 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841263 4409 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841268 4409 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841273 4409 flags.go:64] FLAG: --contention-profiling="false" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841278 4409 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841283 4409 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841288 4409 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841292 4409 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841301 4409 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841306 4409 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841311 4409 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841316 4409 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841320 4409 flags.go:64] FLAG: --enable-server="true" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841325 4409 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841331 4409 flags.go:64] FLAG: --event-burst="100" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841336 4409 flags.go:64] FLAG: --event-qps="50" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841340 4409 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 06:59:04.848792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841346 4409 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841351 4409 flags.go:64] FLAG: --eviction-hard="" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841358 4409 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841362 4409 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841367 4409 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841372 4409 flags.go:64] FLAG: --eviction-soft="" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841377 4409 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841381 4409 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841386 4409 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841391 4409 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841395 4409 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841400 4409 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841404 4409 flags.go:64] FLAG: --feature-gates="" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841410 4409 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841415 4409 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841420 4409 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841425 4409 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841429 4409 flags.go:64] FLAG: --healthz-port="10248" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841435 4409 flags.go:64] FLAG: --help="false" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841439 4409 flags.go:64] FLAG: --hostname-override="ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841445 4409 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841450 4409 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841455 4409 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 06:59:04.849379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841463 4409 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841468 4409 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841473 4409 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841477 4409 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841482 4409 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841486 4409 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841491 4409 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841495 4409 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841500 4409 flags.go:64] FLAG: --kube-reserved="" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841507 4409 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841511 4409 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841516 4409 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841520 4409 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841525 4409 flags.go:64] FLAG: --lock-file="" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841529 4409 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841534 4409 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841538 4409 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841547 4409 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841551 4409 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841556 4409 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841560 4409 flags.go:64] FLAG: --logging-format="text" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841565 4409 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841570 4409 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841575 4409 flags.go:64] FLAG: --manifest-url="" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841579 4409 flags.go:64] FLAG: --manifest-url-header="" Nov 03 06:59:04.849948 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841586 4409 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841590 4409 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841596 4409 flags.go:64] FLAG: --max-pods="110" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841601 4409 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841605 4409 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841610 4409 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841615 4409 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841622 4409 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841627 4409 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841631 4409 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841642 4409 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841646 4409 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841651 4409 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841656 4409 flags.go:64] FLAG: --pod-cidr="" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841661 4409 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841665 4409 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841671 4409 flags.go:64] FLAG: --pods-per-core="0" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841676 4409 flags.go:64] FLAG: --port="10250" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841681 4409 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841686 4409 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-062409b512d164cbf" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841690 4409 flags.go:64] FLAG: --qos-reserved="" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841695 4409 flags.go:64] FLAG: --read-only-port="10255" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841700 4409 flags.go:64] FLAG: --register-node="true" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841705 4409 flags.go:64] FLAG: --register-with-taints="" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841729 4409 flags.go:64] FLAG: --registry-burst="10" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841734 4409 flags.go:64] FLAG: --registry-qps="5" Nov 03 06:59:04.850631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841738 4409 flags.go:64] FLAG: --reserved-cpus="" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841743 4409 flags.go:64] FLAG: --reserved-memory="" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841748 4409 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841753 4409 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841758 4409 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841762 4409 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841767 4409 flags.go:64] FLAG: --runonce="false" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841771 4409 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841776 4409 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841781 4409 flags.go:64] FLAG: --seccomp-default="false" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841786 4409 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841797 4409 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841803 4409 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841809 4409 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841814 4409 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841818 4409 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841823 4409 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841828 4409 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841832 4409 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841837 4409 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841842 4409 flags.go:64] FLAG: --system-cgroups="" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841846 4409 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841857 4409 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841861 4409 flags.go:64] FLAG: --tls-cert-file="" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841866 4409 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 06:59:04.851265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841873 4409 flags.go:64] FLAG: --tls-min-version="" Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841878 4409 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841882 4409 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841887 4409 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841892 4409 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841896 4409 flags.go:64] FLAG: --v="2" Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841902 4409 flags.go:64] FLAG: --version="false" Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841908 4409 flags.go:64] FLAG: --vmodule="" Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841914 4409 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.841920 4409 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842155 4409 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842163 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842168 4409 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842173 4409 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842177 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842181 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842186 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842192 4409 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842198 4409 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842204 4409 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842211 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:04.852168 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842215 4409 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842220 4409 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842224 4409 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842228 4409 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842232 4409 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842237 4409 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842241 4409 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842245 4409 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842250 4409 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842255 4409 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842260 4409 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842264 4409 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842268 4409 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842272 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842277 4409 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842281 4409 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842285 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842289 4409 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842294 4409 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842298 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:04.852811 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842303 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842307 4409 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842311 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842315 4409 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842319 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842323 4409 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842328 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842332 4409 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842336 4409 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842340 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842344 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842350 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842355 4409 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842359 4409 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842364 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842368 4409 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842372 4409 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842376 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842380 4409 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:04.853644 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842385 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842390 4409 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842394 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842399 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842403 4409 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842407 4409 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842411 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842416 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842420 4409 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842424 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842428 4409 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842433 4409 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842437 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842441 4409 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842445 4409 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842449 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842453 4409 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842457 4409 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842461 4409 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842466 4409 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:04.854199 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842470 4409 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842474 4409 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842479 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842483 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842489 4409 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842493 4409 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842503 4409 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842508 4409 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842512 4409 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842516 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842521 4409 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842525 4409 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842529 4409 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842535 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842539 4409 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842543 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842547 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842552 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842556 4409 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:04.854683 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842560 4409 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842567 4409 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842571 4409 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842576 4409 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842581 4409 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842585 4409 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842589 4409 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842593 4409 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842598 4409 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842602 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842606 4409 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842610 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842615 4409 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842619 4409 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842623 4409 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842627 4409 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:04.855320 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842631 4409 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.842675 4409 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842769 4409 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842777 4409 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842783 4409 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842787 4409 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842792 4409 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842796 4409 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842801 4409 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842805 4409 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842813 4409 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842817 4409 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842822 4409 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842826 4409 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842830 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842835 4409 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:04.855750 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842839 4409 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842843 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842847 4409 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842852 4409 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842856 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842861 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842865 4409 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842869 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842874 4409 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842878 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842882 4409 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842886 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842891 4409 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842895 4409 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842899 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842903 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842907 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842911 4409 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842918 4409 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:04.856139 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842922 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842927 4409 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842932 4409 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842936 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842941 4409 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842945 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842949 4409 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842954 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842959 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842963 4409 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842967 4409 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842971 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842974 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842979 4409 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842983 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842987 4409 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842991 4409 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.842995 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843000 4409 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:04.856611 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843004 4409 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843008 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843012 4409 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843017 4409 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843021 4409 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843025 4409 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843029 4409 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843033 4409 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843038 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843043 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843047 4409 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843052 4409 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843056 4409 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843060 4409 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843065 4409 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843069 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843073 4409 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843078 4409 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843082 4409 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843086 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:04.857129 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843091 4409 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843095 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843099 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843103 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843107 4409 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843111 4409 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843118 4409 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843125 4409 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843130 4409 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843134 4409 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843138 4409 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843142 4409 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843146 4409 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843151 4409 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843155 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843159 4409 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843163 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843168 4409 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843173 4409 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:04.857917 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843177 4409 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843181 4409 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843185 4409 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843189 4409 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843193 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843197 4409 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843201 4409 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843205 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843210 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843214 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843222 4409 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843228 4409 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843234 4409 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843238 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.843243 4409 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.846938 4409 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.846953 4409 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.846969 4409 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.846974 4409 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 06:59:04.858534 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847026 4409 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847032 4409 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847035 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847038 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847042 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847045 4409 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847050 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847053 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847056 4409 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847058 4409 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847061 4409 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847063 4409 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847066 4409 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847068 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847071 4409 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847073 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847076 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847078 4409 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847080 4409 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847083 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:04.859111 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847085 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847088 4409 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847091 4409 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847093 4409 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847096 4409 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847099 4409 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847101 4409 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847104 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847106 4409 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847110 4409 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847113 4409 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847116 4409 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847118 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847120 4409 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847123 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847125 4409 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847128 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847130 4409 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847133 4409 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:04.859697 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847135 4409 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847138 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847140 4409 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847143 4409 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847146 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847148 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847151 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847153 4409 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847156 4409 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847158 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847161 4409 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847163 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847166 4409 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847168 4409 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847170 4409 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847173 4409 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847176 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847178 4409 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847181 4409 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:04.860346 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847183 4409 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847186 4409 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847188 4409 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847191 4409 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847194 4409 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847197 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847199 4409 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847201 4409 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847204 4409 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847206 4409 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847209 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847211 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847214 4409 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847216 4409 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847219 4409 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847221 4409 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847224 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847226 4409 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847229 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:04.860989 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847231 4409 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847234 4409 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847236 4409 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847239 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847241 4409 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847244 4409 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847246 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847249 4409 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847251 4409 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847254 4409 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847257 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847259 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847262 4409 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847264 4409 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847267 4409 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847269 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847272 4409 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847275 4409 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847278 4409 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847280 4409 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:04.861578 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847283 4409 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847286 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847288 4409 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847291 4409 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847293 4409 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847296 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847298 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847303 4409 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847306 4409 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.847333 4409 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847480 4409 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847485 4409 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847488 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847490 4409 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847493 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:04.862084 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847495 4409 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847498 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847501 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847503 4409 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847505 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847508 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847510 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847513 4409 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847516 4409 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847518 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847520 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847523 4409 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847525 4409 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847528 4409 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847530 4409 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847532 4409 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847535 4409 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847538 4409 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847540 4409 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:04.862586 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847542 4409 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847545 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847547 4409 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847550 4409 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847552 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847554 4409 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847558 4409 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847561 4409 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847564 4409 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847566 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847569 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847571 4409 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847573 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847576 4409 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847578 4409 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847581 4409 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847583 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847585 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847588 4409 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:04.863392 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847590 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847593 4409 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847596 4409 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847598 4409 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847600 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847603 4409 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847605 4409 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847608 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847610 4409 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847613 4409 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847615 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847618 4409 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847620 4409 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847622 4409 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847625 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847627 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847630 4409 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847633 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847635 4409 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847637 4409 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:04.864144 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847640 4409 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847642 4409 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847644 4409 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847647 4409 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847649 4409 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847652 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847654 4409 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847656 4409 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847659 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847661 4409 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847663 4409 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847666 4409 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847668 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847671 4409 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847673 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847677 4409 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847680 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847682 4409 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847685 4409 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847687 4409 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:04.864873 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847689 4409 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847692 4409 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847694 4409 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847697 4409 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847700 4409 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847702 4409 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847704 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847707 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847724 4409 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847727 4409 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847729 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847732 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847735 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847737 4409 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847740 4409 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847743 4409 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847747 4409 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847750 4409 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847752 4409 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:04.865348 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847755 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847757 4409 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847760 4409 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847763 4409 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.847786 4409 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847815 4409 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847818 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847821 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847825 4409 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847828 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847830 4409 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847833 4409 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847836 4409 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847838 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847841 4409 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847843 4409 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 06:59:04.865864 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847846 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847848 4409 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847851 4409 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847854 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847856 4409 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847858 4409 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847861 4409 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847863 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847866 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847868 4409 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847871 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847873 4409 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847876 4409 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847878 4409 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847880 4409 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847883 4409 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847885 4409 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847887 4409 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847890 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847892 4409 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 06:59:04.866269 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847897 4409 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847900 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847902 4409 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847905 4409 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847909 4409 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847911 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847914 4409 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847916 4409 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847919 4409 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847921 4409 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847924 4409 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847926 4409 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847929 4409 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847931 4409 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847934 4409 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847937 4409 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847939 4409 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847942 4409 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847944 4409 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847947 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 06:59:04.866814 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847949 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847951 4409 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847954 4409 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847956 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847959 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847961 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847964 4409 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847966 4409 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847971 4409 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847974 4409 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847977 4409 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847979 4409 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847984 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847987 4409 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847990 4409 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847993 4409 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847996 4409 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.847999 4409 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848001 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 06:59:04.867472 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848004 4409 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848006 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848009 4409 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848012 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848014 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848016 4409 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848019 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848022 4409 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848024 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848030 4409 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848033 4409 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848035 4409 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848037 4409 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848040 4409 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848042 4409 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848045 4409 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848047 4409 feature_gate.go:386] unrecognized feature gate: Example Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848050 4409 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848052 4409 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848054 4409 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 06:59:04.868100 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848057 4409 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848059 4409 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848062 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848064 4409 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848067 4409 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848070 4409 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848073 4409 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848077 4409 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848080 4409 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848084 4409 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848087 4409 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848089 4409 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848091 4409 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848094 4409 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848096 4409 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:04.848099 4409 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.848146 4409 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.849033 4409 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.849088 4409 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 06:59:04.868568 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.849258 4409 server.go:1014] "Starting client certificate rotation" Nov 03 06:59:04.869036 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.849398 4409 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 06:59:04.869036 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.849437 4409 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 06:53:25 +0000 UTC" deadline="2028-06-03 15:06:38.979011689 +0000 UTC" Nov 03 06:59:04.869036 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.849469 4409 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13880h7m34.129544752s" Nov 03 06:59:04.869036 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.849683 4409 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 06:59:04.869036 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.850732 4409 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 06:59:04.869036 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.852283 4409 log.go:25] "Validated CRI v1 runtime API" Nov 03 06:59:04.869036 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.854351 4409 log.go:25] "Validated CRI v1 image API" Nov 03 06:59:04.869036 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.855119 4409 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 06:59:04.869036 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.856439 4409 fs.go:135] Filesystem UUIDs: map[2a91b4d3-e463-4a40-9ccf-c5458756fa5b:/dev/nvme0n1p3 72604b0a-4219-40dc-9a63-63e626dd03ef:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 06:59:04.869309 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.856458 4409 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/022a29ea035b9e67d182c0eae1955a7cfb3fab7a76d69e2de8d531882bf937be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/022a29ea035b9e67d182c0eae1955a7cfb3fab7a76d69e2de8d531882bf937be/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17aad86066f1d41a3aebd4f5cf755efb5681d5be9c99e3789aeb7a6d0556a431/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17aad86066f1d41a3aebd4f5cf755efb5681d5be9c99e3789aeb7a6d0556a431/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/455ee9c4a945f69306bc3eed3f8d59f87b9c9a30a84caf8e9961bbdc5df9985e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/455ee9c4a945f69306bc3eed3f8d59f87b9c9a30a84caf8e9961bbdc5df9985e/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a7637175a7288f0ee21f9233dd9d1812a75823a3a415497ee2af6f9c92b09e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a7637175a7288f0ee21f9233dd9d1812a75823a3a415497ee2af6f9c92b09e3/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65e91977d8053582f57e74240c8ea6188ccfac691087adf44dc125d9fcea3ef3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65e91977d8053582f57e74240c8ea6188ccfac691087adf44dc125d9fcea3ef3/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71545bab3073c0490e06e49cb45a626b739105f2371f2502405b0b990912057d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71545bab3073c0490e06e49cb45a626b739105f2371f2502405b0b990912057d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/756aba6dcca68596c6f2fd64c7970ea1eba3f709bfff83e8987a2763ff2f12d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/756aba6dcca68596c6f2fd64c7970ea1eba3f709bfff83e8987a2763ff2f12d5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9dd28d3f845f33093c3d5b509fef1d627c672056ce7519537f74a44b99801afc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9dd28d3f845f33093c3d5b509fef1d627c672056ce7519537f74a44b99801afc/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4d96c4ee8a0304c03e8f1d2aec1f5f93f05f428394517bc860855b940456b32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4d96c4ee8a0304c03e8f1d2aec1f5f93f05f428394517bc860855b940456b32/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c319cf8cfa9607a360398f41ee314fcaa87942c7629a2d36f80c193ea818b0f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c319cf8cfa9607a360398f41ee314fcaa87942c7629a2d36f80c193ea818b0f4/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c94a7ec137bf0c0fd9c279bdda4970a2afa45c008c369f16fb6800a8d32a083c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c94a7ec137bf0c0fd9c279bdda4970a2afa45c008c369f16fb6800a8d32a083c/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebb62cd3d5a2fc7f0a61610142cf822d42be28743edc9b1e49113c2d541433ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebb62cd3d5a2fc7f0a61610142cf822d42be28743edc9b1e49113c2d541433ff/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1e02fdc9a47d6890e3bc2b4dcd98300605fabf0256f70ddc018054ee9aa8c25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1e02fdc9a47d6890e3bc2b4dcd98300605fabf0256f70ddc018054ee9aa8c25/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3d86743aa0ad0ccb0d4f17c4e71fc42e5fe4a2865b14ce3c1d8134fd3107274/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3d86743aa0ad0ccb0d4f17c4e71fc42e5fe4a2865b14ce3c1d8134fd3107274/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fc916e0-504f-4aea-85ac-37562156d41e/volumes/kubernetes.io~projected/kube-api-access-9wjn2:{mountpoint:/var/lib/kubelet/pods/0fc916e0-504f-4aea-85ac-37562156d41e/volumes/kubernetes.io~projected/kube-api-access-9wjn2 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a76e41e-99b4-4768-a589-014a65e868ac/volumes/kubernetes.io~projected/kube-api-access-lmkk6:{mountpoint:/var/lib/kubelet/pods/1a76e41e-99b4-4768-a589-014a65e868ac/volumes/kubernetes.io~projected/kube-api-access-lmkk6 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a76e41e-99b4-4768-a589-014a65e868ac/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1a76e41e-99b4-4768-a589-014a65e868ac/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fa0cc98-1f99-4197-9a87-52c9ea4277f9/volumes/kubernetes.io~projected/kube-api-access-9zx74:{mountpoint:/var/lib/kubelet/pods/2fa0cc98-1f99-4197-9a87-52c9ea4277f9/volumes/kubernetes.io~projected/kube-api-access-9zx74 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3378427f-b215-4a67-8a9d-aab1f8b05d8f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/3378427f-b215-4a67-8a9d-aab1f8b05d8f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3378427f-b215-4a67-8a9d-aab1f8b05d8f/volumes/kubernetes.io~projected/kube-api-access-txxt7:{mountpoint:/var/lib/kubelet/pods/3378427f-b215-4a67-8a9d-aab1f8b05d8f/volumes/kubernetes.io~projected/kube-api-access-txxt7 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3378427f-b215-4a67-8a9d-aab1f8b05d8f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/3378427f-b215-4a67-8a9d-aab1f8b05d8f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e9850e7-1c78-455a-b2d0-77971e830f80/volumes/kubernetes.io~projected/kube-api-access-t7gmw:{mountpoint:/var/lib/kubelet/pods/3e9850e7-1c78-455a-b2d0-77971e830f80/volumes/kubernetes.io~projected/kube-api-access-t7gmw major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46730579-83ec-4a31-8a10-759a76d63eac/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/46730579-83ec-4a31-8a10-759a76d63eac/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46730579-83ec-4a31-8a10-759a76d63eac/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/46730579-83ec-4a31-8a10-759a76d63eac/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46730579-83ec-4a31-8a10-759a76d63eac/volumes/kubernetes.io~projected/kube-api-access-wg99r:{mountpoint:/var/lib/kubelet/pods/46730579-83ec-4a31-8a10-759a76d63eac/volumes/kubernetes.io~projected/kube-api-access-wg99r major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6551e74b-f4c1-4be3-8fbf-cf5f262caea4/volumes/kubernetes.io~projected/kube-api-access-xpzg2:{mountpoint:/var/lib/kubelet/pods/6551e74b-f4c1-4be3-8fbf-cf5f262caea4/volumes/kubernetes.io~projected/kube-api-access-xpzg2 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7627e405-9b18-41b2-9fdb-e39eb1ef7e3b/volumes/kubernetes.io~projected/kube-api-access-b2bnh:{mountpoint:/var/lib/kubelet/pods/7627e405-9b18-41b2-9fdb-e39eb1ef7e3b/volumes/kubernetes.io~projected/kube-api-access-b2bnh major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d5f4689-033d-4aa8-8603-a56cbdd449cb/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8d5f4689-033d-4aa8-8603-a56cbdd449cb/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d832341-de73-4a7b-9ce0-c41bbe655035/volumes/kubernetes.io~projected/kube-api-access-mwk9n:{mountpoint:/var/lib/kubelet/pods/8d832341-de73-4a7b-9ce0-c41bbe655035/volumes/kubernetes.io~projected/kube-api-access-mwk9n major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fbdc7ee-c57c-4a65-ba85-b20706762749/volumes/kubernetes.io~projected/kube-api-access-7qk6d:{mountpoint:/var/lib/kubelet/pods/9fbdc7ee-c57c-4a65-ba85-b20706762749/volumes/kubernetes.io~projected/kube-api-access-7qk6d major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d399cdee-b235-47b8-96e5-1687506510d0/volumes/kubernetes.io~projected/kube-api-access-4jtj6:{mountpoint:/var/lib/kubelet/pods/d399cdee-b235-47b8-96e5-1687506510d0/volumes/kubernetes.io~projected/kube-api-access-4jtj6 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eaee23aa-3918-44a3-a35c-7a77133e45c7/volumes/kubernetes.io~projected/kube-api-access-f2z4q:{mountpoint:/var/lib/kubelet/pods/eaee23aa-3918-44a3-a35c-7a77133e45c7/volumes/kubernetes.io~projected/kube-api-access-f2z4q major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eff68601-5319-4dc8-93c6-16de197bd812/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/eff68601-5319-4dc8-93c6-16de197bd812/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3/volumes/kubernetes.io~projected/kube-api-access-fq9tc:{mountpoint:/var/lib/kubelet/pods/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3/volumes/kubernetes.io~projected/kube-api-access-fq9tc major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe326448-785b-4e7a-a189-76298b83c9b3/volumes/kubernetes.io~projected/kube-api-access-jxlxl:{mountpoint:/var/lib/kubelet/pods/fe326448-785b-4e7a-a189-76298b83c9b3/volumes/kubernetes.io~projected/kube-api-access-jxlxl major:0 minor:52 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3f1ddfb8723f72c9e3e67c53e0e7ef2fc0c91a757c87e81015eecefa0505fff9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a6d6ce73f6deb5eff8249dd4f87efa071e9b391c776493523b7940268804200d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4f278fc2e8b94b3dcea5d579ddb9147785e446ca6b380ff69467ecb7608f6bae/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/c7e9fb749f9cb8990f595498bf098e193ff85091668b9d28c62c867aa971e1fb/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/cc40f9582b38f807ff197280ee8a5830135a6bef50ad4c70697b6e92f1f26f9d/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/dad3589c7b66236e3d174d49dbc51f58ca92e71b4a66270380264a0f44156b53/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/2e073924fe66625eb1a96eb566eea2bc2fcdf33c0f8c1480cbacef9ef1f6cae1/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f91996728ccaa9701049691013ac07f5d1607539c379f3211d67e0d411a17861/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/436f5cc74c82b1e943214755848d29b5ff1c3a220eb869754e1665547098fb72/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/64f4a91b4b2efc61d047b72cd8add83c15c540fcfe986295b53b60fcc67b08b9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/72d5c23033fe4597ae0f2759a2cab8a701d1fec45e27525a8265a3ac7bee0904/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/8d9f1ab3a6dde8e042bfe11ee9b1efdf11a644fe65c75d205b9536794d51345e/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/5593d50f9566cf8921fc78161f47a74f9956e75c531b9a9a589b207bfed09a11/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e9cd0e301f83564bd51eb59db40d1b08d354d0180d0d6c0df4d59fe22d5af6c6/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/372acda9cc1ffac2da40968ca94a449a23fdf18dd4966882afdddb536c1b6225/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/769398d8eb943d4e53ea92e7075f853237482d2fa227ae8abfd02430f272a353/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/0e4f56ebf7a8cf1831007b4fe7a93ae0de5d27454947acebf78eb4cbac227406/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1de238a49eaa2455bccbf8255797893a112509146ca258ebfb7861cbfa9f877e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e8f116abe387490ffc844b5b5873f6eae325b4308e6e2c8fa27430fef5b26b67/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e0db340e6249781dc7523c12e0a95593d8345daf0c29ab82b87c904394ff0740/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d219770062055bf8afadbca93b04612968e6e794d39b9a2e369cc6fe51e2f468/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/102bca618a403d72b568f646afe5fc51c65930474453f460db1cc5e8a7e6d8a7/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/1b444c7b89ccf2f8ecd81d6e00d5010aebed1af07072c4b1e38af23d093556b5/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/e6fa5da07f5c17a07d5f0a9cd99e4a59ddc7e6f1d95d198d2d039718bcf39a4c/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/c26a40363a981239a03647c067299d97083d43017c0bda8264e711838298b0e4/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/1c6ae4b5b03a0521eb6611be0e3448c7c436cb8fdf005d0e69e97d34da3b037a/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/15c4aa7d8ba002f240da4422fc9222b67b164fab2550aa91beb4b67f0ed1c304/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/f16b5cab4742a9a38352c7f0ddb755933d070f43b1fe97345038d46f55ea8552/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/9ba0f4f4dca84d4c4f1b14b8cb6f5bfb119fbf6d4a2b5ba88f7ddc61f827073d/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-63:{mountpoint:/var/lib/containers/storage/overlay/6c382010a62622fbbcf807ac47df712447ff3e189798ac8911671671a2f06b51/merged major:0 minor:63 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/b0e01d03a384fe9f979e23f3df579997ccb559ae3e450aaa7bd277002eab2884/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/9e7cd901dc9aa5ff5982b57b1594516059e8dc6ba655597240c5e964b4c5d0d0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ff327c1e6060a5fe6615a12909fd101c64d1bc7a897b0f1d6ec181e89ad7c220/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3ff4a343d9e23058dc6db723a37ac62199bc315cd0e20edd2581209438d46ee0/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a0e183c29764771ef2dce38a6fe0ed8a6a4b5c24591e436c2b6a01dcaa684b1b/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/bcee75ec3327bb2cc78c6f67bec9084b69872f2edd5fc962f49e952c95284301/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/22e2e55fb7072e63887aeab5da536c0df9fd9b447e8ed65b5dd407a3c8ef446a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d7bec0cdd6813b767e2f971223f13973f169a97ca789c03a4bc5c07ac382c0b9/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9665c3220c369ee2036d8e99b218e355a954ff8c7aab0a044135d53226b9f182/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.862006 4409 manager.go:217] Machine: {Timestamp:2026-11-03 06:59:04.861543672 +0000 UTC m=+0.054085414 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100371 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2df219d8404368faa465956e378006 SystemUUID:ec2df219-d840-4368-faa4-65956e378006 BootID:06df53c6-48e5-4686-b627-66894f6f5cd5 Filesystems:[{Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fc916e0-504f-4aea-85ac-37562156d41e/volumes/kubernetes.io~projected/kube-api-access-9wjn2 DeviceMajor:0 DeviceMinor:209 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1e02fdc9a47d6890e3bc2b4dcd98300605fabf0256f70ddc018054ee9aa8c25/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/022a29ea035b9e67d182c0eae1955a7cfb3fab7a76d69e2de8d531882bf937be/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46730579-83ec-4a31-8a10-759a76d63eac/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8d832341-de73-4a7b-9ce0-c41bbe655035/volumes/kubernetes.io~projected/kube-api-access-mwk9n DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a7637175a7288f0ee21f9233dd9d1812a75823a3a415497ee2af6f9c92b09e3/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3d86743aa0ad0ccb0d4f17c4e71fc42e5fe4a2865b14ce3c1d8134fd3107274/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46730579-83ec-4a31-8a10-759a76d63eac/volumes/kubernetes.io~projected/kube-api-access-wg99r DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a76e41e-99b4-4768-a589-014a65e868ac/volumes/kubernetes.io~projected/kube-api-access-lmkk6 DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3378427f-b215-4a67-8a9d-aab1f8b05d8f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7627e405-9b18-41b2-9fdb-e39eb1ef7e3b/volumes/kubernetes.io~projected/kube-api-access-b2bnh DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3378427f-b215-4a67-8a9d-aab1f8b05d8f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-63 DeviceMajor:0 DeviceMinor:63 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d399cdee-b235-47b8-96e5-1687506510d0/volumes/kubernetes.io~projected/kube-api-access-4jtj6 DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c319cf8cfa9607a360398f41ee314fcaa87942c7629a2d36f80c193ea818b0f4/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65e91977d8053582f57e74240c8ea6188ccfac691087adf44dc125d9fcea3ef3/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c94a7ec137bf0c0fd9c279bdda4970a2afa45c008c369f16fb6800a8d32a083c/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d5f4689-033d-4aa8-8603-a56cbdd449cb/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:212 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/eaee23aa-3918-44a3-a35c-7a77133e45c7/volumes/kubernetes.io~projected/kube-api-access-f2z4q DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3e9850e7-1c78-455a-b2d0-77971e830f80/volumes/kubernetes.io~projected/kube-api-access-t7gmw DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/756aba6dcca68596c6f2fd64c7970ea1eba3f709bfff83e8987a2763ff2f12d5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fbdc7ee-c57c-4a65-ba85-b20706762749/volumes/kubernetes.io~projected/kube-api-access-7qk6d DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17aad86066f1d41a3aebd4f5cf755efb5681d5be9c99e3789aeb7a6d0556a431/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fa0cc98-1f99-4197-9a87-52c9ea4277f9/volumes/kubernetes.io~projected/kube-api-access-9zx74 DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/455ee9c4a945f69306bc3eed3f8d59f87b9c9a30a84caf8e9961bbdc5df9985e/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3/volumes/kubernetes.io~projected/kube-api-access-fq9tc DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe326448-785b-4e7a-a189-76298b83c9b3/volumes/kubernetes.io~projected/kube-api-access-jxlxl DeviceMajor:0 DeviceMinor:52 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebb62cd3d5a2fc7f0a61610142cf822d42be28743edc9b1e49113c2d541433ff/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a76e41e-99b4-4768-a589-014a65e868ac/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6551e74b-f4c1-4be3-8fbf-cf5f262caea4/volumes/kubernetes.io~projected/kube-api-access-xpzg2 DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9dd28d3f845f33093c3d5b509fef1d627c672056ce7519537f74a44b99801afc/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71545bab3073c0490e06e49cb45a626b739105f2371f2502405b0b990912057d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eff68601-5319-4dc8-93c6-16de197bd812/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46730579-83ec-4a31-8a10-759a76d63eac/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3378427f-b215-4a67-8a9d-aab1f8b05d8f/volumes/kubernetes.io~projected/kube-api-access-txxt7 DeviceMajor:0 DeviceMinor:51 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4d96c4ee8a0304c03e8f1d2aec1f5f93f05f428394517bc860855b940456b32/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:022a29ea035b9e6 MacAddress:96:ba:f8:8c:d2:70 Speed:10000 Mtu:8901} {Name:65e91977d805358 MacAddress:82:88:3a:77:94:80 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:01:88:bf:2b:a9 Speed:0 Mtu:9001} {Name:br-int MacAddress:ea:f3:b4:da:4e:a0 Speed:0 Mtu:8901} {Name:c319cf8cfa9607a MacAddress:4e:76:d7:96:bf:87 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:01:88:bf:2b:a9 Speed:0 Mtu:9001} {Name:f1e02fdc9a47d68 MacAddress:82:46:47:88:13:0c Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:1a:89:ef:f4:7e:0c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8a:b9:3b:39:f9:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.862216 4409 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.862254 4409 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.862567 4409 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.862591 4409 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-121.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.862791 4409 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.862804 4409 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.862818 4409 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.862844 4409 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.862950 4409 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.863151 4409 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.863278 4409 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.863321 4409 kubelet.go:493] "Attempting to sync node with API server" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.863335 4409 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.863348 4409 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.863360 4409 kubelet.go:405] "Adding apiserver pod source" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.863370 4409 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.864110 4409 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.864122 4409 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.864550 4409 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866412 4409 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866420 4409 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866513 4409 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866697 4409 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866747 4409 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866843 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866860 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866870 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866877 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866884 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866891 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866899 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866906 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866914 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866919 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866927 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866936 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866957 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 06:59:04.869440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.866964 4409 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 06:59:04.870907 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.870104 4409 server.go:1263] "Started kubelet" Nov 03 06:59:04.870907 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.870345 4409 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 06:59:04.870619 ip-10-0-138-121 systemd[1]: Started Kubernetes Kubelet. Nov 03 06:59:04.872043 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.870940 4409 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 06:59:04.872043 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.871576 4409 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 06:59:04.872043 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.871892 4409 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 06:59:04.874366 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.873390 4409 server.go:317] "Adding debug handlers to kubelet server" Nov 03 06:59:04.876309 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.876289 4409 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 06:59:04.876395 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.876323 4409 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 06:59:04.876395 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.876368 4409 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 06:53:26 +0000 UTC" deadline="2028-05-28 07:19:19.956776134 +0000 UTC" Nov 03 06:59:04.876501 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.876391 4409 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13728h20m15.080387858s" Nov 03 06:59:04.876501 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.876428 4409 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 06:59:04.876501 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.876436 4409 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 06:59:04.876638 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.876533 4409 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 06:59:04.877529 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:04.877052 4409 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-121.us-west-1.compute.internal\" not found" Nov 03 06:59:04.878152 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878101 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaee23aa-3918-44a3-a35c-7a77133e45c7" volumeName="kubernetes.io/configmap/eaee23aa-3918-44a3-a35c-7a77133e45c7-iptables-alerter-script" seLinuxMountContext="" Nov 03 06:59:04.878248 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878154 4409 factory.go:55] Registering systemd factory Nov 03 06:59:04.878248 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878171 4409 factory.go:223] Registration of the systemd container factory successfully Nov 03 06:59:04.878248 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878151 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3" volumeName="kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc" seLinuxMountContext="" Nov 03 06:59:04.878248 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878241 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9850e7-1c78-455a-b2d0-77971e830f80" volumeName="kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878265 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46730579-83ec-4a31-8a10-759a76d63eac" volumeName="kubernetes.io/empty-dir/46730579-83ec-4a31-8a10-759a76d63eac-etc-tuned" seLinuxMountContext="" Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878281 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46730579-83ec-4a31-8a10-759a76d63eac" volumeName="kubernetes.io/projected/46730579-83ec-4a31-8a10-759a76d63eac-kube-api-access-wg99r" seLinuxMountContext="" Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878302 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d399cdee-b235-47b8-96e5-1687506510d0" volumeName="kubernetes.io/configmap/d399cdee-b235-47b8-96e5-1687506510d0-config-volume" seLinuxMountContext="" Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878316 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d399cdee-b235-47b8-96e5-1687506510d0" volumeName="kubernetes.io/projected/d399cdee-b235-47b8-96e5-1687506510d0-kube-api-access-4jtj6" seLinuxMountContext="" Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878329 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9850e7-1c78-455a-b2d0-77971e830f80" volumeName="kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878353 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9850e7-1c78-455a-b2d0-77971e830f80" volumeName="kubernetes.io/projected/3e9850e7-1c78-455a-b2d0-77971e830f80-kube-api-access-t7gmw" seLinuxMountContext="" Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878365 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6551e74b-f4c1-4be3-8fbf-cf5f262caea4" volumeName="kubernetes.io/projected/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-kube-api-access-xpzg2" seLinuxMountContext="" Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878379 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7627e405-9b18-41b2-9fdb-e39eb1ef7e3b" volumeName="kubernetes.io/projected/7627e405-9b18-41b2-9fdb-e39eb1ef7e3b-kube-api-access-b2bnh" seLinuxMountContext="" Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878401 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fbdc7ee-c57c-4a65-ba85-b20706762749" volumeName="kubernetes.io/projected/9fbdc7ee-c57c-4a65-ba85-b20706762749-kube-api-access-7qk6d" seLinuxMountContext="" Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878414 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eff68601-5319-4dc8-93c6-16de197bd812" volumeName="kubernetes.io/secret/eff68601-5319-4dc8-93c6-16de197bd812-agent-certs" seLinuxMountContext="" Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878424 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe326448-785b-4e7a-a189-76298b83c9b3" volumeName="kubernetes.io/projected/fe326448-785b-4e7a-a189-76298b83c9b3-kube-api-access-jxlxl" seLinuxMountContext="" Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878437 4409 factory.go:153] Registering CRI-O factory Nov 03 06:59:04.878451 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878454 4409 factory.go:223] Registration of the crio container factory successfully Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878459 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fc916e0-504f-4aea-85ac-37562156d41e" volumeName="kubernetes.io/empty-dir/0fc916e0-504f-4aea-85ac-37562156d41e-data-volume" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878499 4409 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878523 4409 factory.go:103] Registering Raw factory Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878541 4409 manager.go:1199] Started watching for new ooms in manager Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878473 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fc916e0-504f-4aea-85ac-37562156d41e" volumeName="kubernetes.io/projected/0fc916e0-504f-4aea-85ac-37562156d41e-kube-api-access-9wjn2" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878599 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3378427f-b215-4a67-8a9d-aab1f8b05d8f" volumeName="kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovnkube-script-lib" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878622 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3378427f-b215-4a67-8a9d-aab1f8b05d8f" volumeName="kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovnkube-config" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878637 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3378427f-b215-4a67-8a9d-aab1f8b05d8f" volumeName="kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-env-overrides" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878667 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3378427f-b215-4a67-8a9d-aab1f8b05d8f" volumeName="kubernetes.io/secret/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878682 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9850e7-1c78-455a-b2d0-77971e830f80" volumeName="kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-cni-binary-copy" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878731 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaee23aa-3918-44a3-a35c-7a77133e45c7" volumeName="kubernetes.io/projected/eaee23aa-3918-44a3-a35c-7a77133e45c7-kube-api-access-f2z4q" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878748 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fa0cc98-1f99-4197-9a87-52c9ea4277f9" volumeName="kubernetes.io/projected/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-kube-api-access-9zx74" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878760 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6551e74b-f4c1-4be3-8fbf-cf5f262caea4" volumeName="kubernetes.io/configmap/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-daemon-config" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878773 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eff68601-5319-4dc8-93c6-16de197bd812" volumeName="kubernetes.io/configmap/eff68601-5319-4dc8-93c6-16de197bd812-konnectivity-ca" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878788 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d832341-de73-4a7b-9ce0-c41bbe655035" volumeName="kubernetes.io/projected/8d832341-de73-4a7b-9ce0-c41bbe655035-kube-api-access-mwk9n" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878804 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a76e41e-99b4-4768-a589-014a65e868ac" volumeName="kubernetes.io/projected/1a76e41e-99b4-4768-a589-014a65e868ac-kube-api-access-lmkk6" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878816 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5f4689-033d-4aa8-8603-a56cbdd449cb" volumeName="kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878836 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fbdc7ee-c57c-4a65-ba85-b20706762749" volumeName="kubernetes.io/configmap/9fbdc7ee-c57c-4a65-ba85-b20706762749-serviceca" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878852 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fa0cc98-1f99-4197-9a87-52c9ea4277f9" volumeName="kubernetes.io/empty-dir/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-tmp-dir" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878864 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3378427f-b215-4a67-8a9d-aab1f8b05d8f" volumeName="kubernetes.io/projected/3378427f-b215-4a67-8a9d-aab1f8b05d8f-kube-api-access-txxt7" seLinuxMountContext="" Nov 03 06:59:04.878869 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878876 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d399cdee-b235-47b8-96e5-1687506510d0" volumeName="kubernetes.io/empty-dir/d399cdee-b235-47b8-96e5-1687506510d0-tmp-dir" seLinuxMountContext="" Nov 03 06:59:04.879400 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878887 4409 manager.go:319] Starting recovery of all containers Nov 03 06:59:04.879813 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.878888 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a76e41e-99b4-4768-a589-014a65e868ac" volumeName="kubernetes.io/secret/1a76e41e-99b4-4768-a589-014a65e868ac-cert" seLinuxMountContext="" Nov 03 06:59:04.879910 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.879839 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46730579-83ec-4a31-8a10-759a76d63eac" volumeName="kubernetes.io/empty-dir/46730579-83ec-4a31-8a10-759a76d63eac-tmp" seLinuxMountContext="" Nov 03 06:59:04.879910 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.879865 4409 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6551e74b-f4c1-4be3-8fbf-cf5f262caea4" volumeName="kubernetes.io/configmap/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-cni-binary-copy" seLinuxMountContext="" Nov 03 06:59:04.879910 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.879879 4409 reconstruct.go:97] "Volume reconstruction finished" Nov 03 06:59:04.879910 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.879889 4409 reconciler.go:29] "Reconciler: start to sync state" Nov 03 06:59:04.888691 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.886039 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 06:59:04.888691 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.886324 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 06:59:04.903678 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.903645 4409 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 06:59:04.904937 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.904918 4409 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 06:59:04.905038 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.904965 4409 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 06:59:04.905038 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.904981 4409 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 06:59:04.905038 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:04.905010 4409 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 06:59:04.920769 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.920750 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 06:59:04.921083 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.921062 4409 generic.go:356] "Generic (PLEG): container finished" podID="3378427f-b215-4a67-8a9d-aab1f8b05d8f" containerID="620610d68e5af2f05411b0637db646d58e3676d95339a809e80dd89d665a2a99" exitCode=1 Nov 03 06:59:04.932983 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.932960 4409 generic.go:356] "Generic (PLEG): container finished" podID="3e9850e7-1c78-455a-b2d0-77971e830f80" containerID="8259cceb94ed89a2d67fc53a8b24fb88493d6932efc57596689d941f15d2b3c3" exitCode=0 Nov 03 06:59:04.933091 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.932983 4409 generic.go:356] "Generic (PLEG): container finished" podID="3e9850e7-1c78-455a-b2d0-77971e830f80" containerID="a293e9b889349b49c758759a5735efb18b264439134b8affbaa7762b86090bc9" exitCode=0 Nov 03 06:59:04.933091 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.932994 4409 generic.go:356] "Generic (PLEG): container finished" podID="3e9850e7-1c78-455a-b2d0-77971e830f80" containerID="902485287b93715ebabc966956bb67a886ff1a6519b161e106a76d6031cd98a7" exitCode=0 Nov 03 06:59:04.933091 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.933001 4409 generic.go:356] "Generic (PLEG): container finished" podID="3e9850e7-1c78-455a-b2d0-77971e830f80" containerID="9ba7809bc91c7ea0179d7eabdc4b555a04fb9746fc3f77ba198c19366fd2fcd1" exitCode=0 Nov 03 06:59:04.933091 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.933006 4409 generic.go:356] "Generic (PLEG): container finished" podID="3e9850e7-1c78-455a-b2d0-77971e830f80" containerID="a1968da4a34f6ecbf768237625ace6f404ce71f008e0e2b72970a8d59dcc9a54" exitCode=0 Nov 03 06:59:04.933091 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.933011 4409 generic.go:356] "Generic (PLEG): container finished" podID="3e9850e7-1c78-455a-b2d0-77971e830f80" containerID="e37c2cafcd5016a24b8b0f7427204d32e7c538787b06711376b69359457e9650" exitCode=0 Nov 03 06:59:04.941063 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.941040 4409 generic.go:356] "Generic (PLEG): container finished" podID="c99460852de5d8046fca947f507d3971" containerID="dfd07c97b82fef8c06e475b8ca472ee9f567ad3228198a1ee86d88f5b5165d8e" exitCode=0 Nov 03 06:59:04.946543 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.946529 4409 manager.go:324] Recovery completed Nov 03 06:59:04.959168 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.959150 4409 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:04.960362 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.960347 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:04.960460 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.960379 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:04.960460 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.960395 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:04.961703 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.961687 4409 cpu_manager.go:226] "Starting" policy="none" Nov 03 06:59:04.961703 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.961699 4409 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 06:59:04.961703 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.961703 4409 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:04.961875 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.961731 4409 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 06:59:04.961875 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.961842 4409 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 06:59:04.961875 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.961852 4409 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 06:59:04.961875 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.961866 4409 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 06:59:04.961875 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.961871 4409 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 06:59:04.961875 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.961877 4409 policy_none.go:50] "Start" Nov 03 06:59:04.962053 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.961886 4409 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 06:59:04.962053 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.961895 4409 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 06:59:04.962053 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.962010 4409 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 06:59:04.962053 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.962017 4409 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 06:59:04.962053 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.962022 4409 policy_none.go:44] "Start" Nov 03 06:59:04.962756 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.962741 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:04.962816 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.962775 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:04.962816 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.962792 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:04.966008 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.965956 4409 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 06:59:04.968495 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.968483 4409 manager.go:341] "Starting Device Plugin manager" Nov 03 06:59:04.968580 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:04.968512 4409 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 06:59:04.968580 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.968521 4409 server.go:87] "Starting device plugin registration server" Nov 03 06:59:04.968797 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.968785 4409 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 06:59:04.968858 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.968798 4409 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 06:59:04.968904 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.968875 4409 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 06:59:04.969503 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.968971 4409 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 06:59:04.969503 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.969027 4409 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 06:59:04.969503 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.969036 4409 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 06:59:04.969503 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.969099 4409 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T06:59:04.969011531Z","UUID":"46875a77-2dba-4278-8654-14c574e4ee83","Handler":null,"Name":"","Endpoint":""} Nov 03 06:59:04.970808 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:04.970708 4409 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 06:59:04.970808 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:04.970789 4409 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-121.us-west-1.compute.internal\" not found" Nov 03 06:59:04.972393 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.971665 4409 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 06:59:04.972393 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:04.971702 4409 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 06:59:05.005883 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.005280 4409 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal"] Nov 03 06:59:05.005965 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.005953 4409 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:05.006822 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.006802 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:05.006895 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.006833 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:05.006895 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.006844 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:05.007074 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.007064 4409 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:05.007203 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.007188 4409 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:05.008050 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.008023 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:05.008122 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.008057 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:05.008122 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.008063 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:05.008122 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.008072 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:05.008122 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.008084 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:05.008122 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.008094 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:05.008267 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:05.008201 4409 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-121.us-west-1.compute.internal\" not found" node="ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.008322 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.008299 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" event={"ID":"305af933eb9c6382ff5eabda70a436a6","Type":"ContainerStarted","Data":"8bb31c6c3ec9aec7b2ef09d1261e7cd4d4134b13e98605ac4c6e6848e3d574f0"} Nov 03 06:59:05.008359 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.008330 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" event={"ID":"305af933eb9c6382ff5eabda70a436a6","Type":"ContainerStarted","Data":"4a7637175a7288f0ee21f9233dd9d1812a75823a3a415497ee2af6f9c92b09e3"} Nov 03 06:59:05.008389 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.008382 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" event={"ID":"c99460852de5d8046fca947f507d3971","Type":"ContainerStarted","Data":"f6e5cb05cd3fa4c6752b527eaaf12dff7095b9c4642c4ba58f8ced498edcc9cf"} Nov 03 06:59:05.008416 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.008394 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" event={"ID":"c99460852de5d8046fca947f507d3971","Type":"ContainerDied","Data":"dfd07c97b82fef8c06e475b8ca472ee9f567ad3228198a1ee86d88f5b5165d8e"} Nov 03 06:59:05.008416 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.008406 4409 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:05.008497 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.008406 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" event={"ID":"c99460852de5d8046fca947f507d3971","Type":"ContainerStarted","Data":"455ee9c4a945f69306bc3eed3f8d59f87b9c9a30a84caf8e9961bbdc5df9985e"} Nov 03 06:59:05.009321 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.009305 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:05.009426 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.009335 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:05.009426 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.009349 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:05.009543 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:05.009497 4409 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-121.us-west-1.compute.internal\" not found" node="ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.069587 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.069567 4409 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 06:59:05.071753 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.071740 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 06:59:05.071815 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.071766 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 06:59:05.071815 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.071775 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 06:59:05.071815 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.071789 4409 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.078153 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.078138 4409 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 06:59:05.127473 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.127452 4409 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:59:05.127800 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.127785 4409 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:59:05.127877 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.127801 4409 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:59:05.127877 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.127873 4409 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 06:59:05.177840 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.177820 4409 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.178942 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.178926 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/305af933eb9c6382ff5eabda70a436a6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"305af933eb9c6382ff5eabda70a436a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.179006 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.178956 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c99460852de5d8046fca947f507d3971-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"c99460852de5d8046fca947f507d3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.179006 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.178983 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c99460852de5d8046fca947f507d3971-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"c99460852de5d8046fca947f507d3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.221211 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.221161 4409 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.221297 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.221218 4409 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.221699 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.221685 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeNotReady" Nov 03 06:59:05.221770 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.221734 4409 setters.go:546] "Node became not ready" node="ip-10-0-138-121.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T06:59:05Z","lastTransitionTime":"2026-11-03T06:59:05Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 06:59:05.228521 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.228500 4409 nodeinfomanager.go:471] existing CSINode "ip-10-0-138-121.us-west-1.compute.internal" is owned by different node (oldNodeID="285565a0-26bb-4cc7-9596-86b706d2864b", newNodeID=""), cleaning up... Nov 03 06:59:05.239338 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.239319 4409 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 06:59:05.239410 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:05.239353 4409 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.239410 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.239363 4409 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.279320 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.279302 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/305af933eb9c6382ff5eabda70a436a6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"305af933eb9c6382ff5eabda70a436a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.279386 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.279325 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c99460852de5d8046fca947f507d3971-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"c99460852de5d8046fca947f507d3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.279386 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.279344 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c99460852de5d8046fca947f507d3971-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"c99460852de5d8046fca947f507d3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.279386 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.279383 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c99460852de5d8046fca947f507d3971-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"c99460852de5d8046fca947f507d3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.279477 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.279407 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/305af933eb9c6382ff5eabda70a436a6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"305af933eb9c6382ff5eabda70a436a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.279477 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.279416 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c99460852de5d8046fca947f507d3971-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" (UID: \"c99460852de5d8046fca947f507d3971\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.301071 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.301055 4409 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 06:59:05.301137 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:05.301087 4409 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.863793 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.863764 4409 apiserver.go:52] "Watching apiserver" Nov 03 06:59:05.944127 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.944105 4409 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:05.944259 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:05.944236 4409 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:06.019352 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.019322 4409 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 06:59:06.019352 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.019346 4409 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 06:59:06.019525 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.019372 4409 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:06.020049 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.019779 4409 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal" Nov 03 06:59:06.020049 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.019818 4409 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 06:59:06.021197 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.021162 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-121.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal","openshift-multus/multus-rww5p","openshift-ingress-canary/ingress-canary-kkrxc","openshift-dns/dns-default-htqcm","openshift-multus/multus-additional-cni-plugins-4ndld","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v","openshift-network-diagnostics/network-check-target-9gjvc","kube-system/global-pull-secret-syncer-zhpcz","kube-system/konnectivity-agent-nj9qb","openshift-image-registry/node-ca-nn9v4","openshift-network-operator/iptables-alerter-s5f7z","openshift-cluster-node-tuning-operator/tuned-6f246","openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw","openshift-multus/network-metrics-daemon-cd7lc","openshift-ovn-kubernetes/ovnkube-node-l4ffc","openshift-dns/node-resolver-mwws4","openshift-insights/insights-runtime-extractor-wqbfp"] Nov 03 06:59:06.025660 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.025642 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:59:06.025997 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.025982 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:06.026974 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.026956 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.077756 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.077736 4409 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 06:59:06.077975 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.077956 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 06:59:06.077975 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.077971 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 06:59:06.078149 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.078135 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 06:59:06.078223 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.078177 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5znfn\"" Nov 03 06:59:06.078223 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.078204 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 06:59:06.078319 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.078240 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tq2gc\"" Nov 03 06:59:06.078370 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.078317 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 06:59:06.078370 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.078345 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 06:59:06.078531 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.078501 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 06:59:06.078669 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.078637 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 06:59:06.078912 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.078896 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8c889\"" Nov 03 06:59:06.078998 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.078927 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 06:59:06.078998 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.078986 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 06:59:06.079101 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.078997 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 06:59:06.079101 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079024 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44mbd\"" Nov 03 06:59:06.079101 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079082 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 06:59:06.079101 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079082 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 06:59:06.079101 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079093 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 06:59:06.079265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079106 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 06:59:06.079265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079140 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 06:59:06.079265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079183 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dmgt\"" Nov 03 06:59:06.079265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079206 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 06:59:06.079265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079206 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 06:59:06.079265 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079244 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 06:59:06.079436 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079277 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 06:59:06.079436 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079289 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 06:59:06.079436 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079297 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l9lht\"" Nov 03 06:59:06.079436 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079305 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 06:59:06.079436 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079330 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 06:59:06.079436 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079332 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:59:06.079436 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079357 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-br79x\"" Nov 03 06:59:06.079436 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.079415 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-htlhr\"" Nov 03 06:59:06.083374 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083356 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-cnibin\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.083447 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083390 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpzg2\" (UniqueName: \"kubernetes.io/projected/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-kube-api-access-xpzg2\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.083447 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083415 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bnh\" (UniqueName: \"kubernetes.io/projected/7627e405-9b18-41b2-9fdb-e39eb1ef7e3b-kube-api-access-b2bnh\") pod \"migrator-66df5d496b-2tcnw\" (UID: \"7627e405-9b18-41b2-9fdb-e39eb1ef7e3b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw" Nov 03 06:59:06.083447 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083443 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.083595 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083471 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.083595 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083499 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-conf-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.083595 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083529 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fc916e0-504f-4aea-85ac-37562156d41e-data-volume\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.083763 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083683 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovn-node-metrics-cert\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.083763 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083734 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg99r\" (UniqueName: \"kubernetes.io/projected/46730579-83ec-4a31-8a10-759a76d63eac-kube-api-access-wg99r\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.083858 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083765 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-cni-bin\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.083858 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083800 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.083858 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083770 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fc916e0-504f-4aea-85ac-37562156d41e-data-volume\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.084011 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083854 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jtj6\" (UniqueName: \"kubernetes.io/projected/d399cdee-b235-47b8-96e5-1687506510d0-kube-api-access-4jtj6\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:06.084011 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083923 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjn2\" (UniqueName: \"kubernetes.io/projected/0fc916e0-504f-4aea-85ac-37562156d41e-kube-api-access-9wjn2\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.084011 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083958 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eff68601-5319-4dc8-93c6-16de197bd812-konnectivity-ca\") pod \"konnectivity-agent-nj9qb\" (UID: \"eff68601-5319-4dc8-93c6-16de197bd812\") " pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:59:06.084011 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.083985 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-lib-modules\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.084011 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084009 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-k8s-cni-cncf-io\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.084281 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084046 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.084281 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084101 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:59:06.084281 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084122 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.084281 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084124 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-node-log\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.084281 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084149 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-registration-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.084281 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084180 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8d5f4689-033d-4aa8-8603-a56cbdd449cb-dbus\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:59:06.084281 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084197 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmkk6\" (UniqueName: \"kubernetes.io/projected/1a76e41e-99b4-4768-a589-014a65e868ac-kube-api-access-lmkk6\") pod \"ingress-canary-kkrxc\" (UID: \"1a76e41e-99b4-4768-a589-014a65e868ac\") " pod="openshift-ingress-canary/ingress-canary-kkrxc" Nov 03 06:59:06.084281 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084212 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.084281 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084257 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7gmw\" (UniqueName: \"kubernetes.io/projected/3e9850e7-1c78-455a-b2d0-77971e830f80-kube-api-access-t7gmw\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.084635 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084309 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d399cdee-b235-47b8-96e5-1687506510d0-tmp-dir\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:06.084635 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084329 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.084635 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084371 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eff68601-5319-4dc8-93c6-16de197bd812-agent-certs\") pod \"konnectivity-agent-nj9qb\" (UID: \"eff68601-5319-4dc8-93c6-16de197bd812\") " pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:59:06.084635 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084387 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d399cdee-b235-47b8-96e5-1687506510d0-tmp-dir\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:06.084635 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084414 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovn-node-metrics-cert\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.084635 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084419 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2z4q\" (UniqueName: \"kubernetes.io/projected/eaee23aa-3918-44a3-a35c-7a77133e45c7-kube-api-access-f2z4q\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:59:06.084635 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084456 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-kubernetes\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.084635 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084503 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-cni-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.084635 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084614 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084642 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084669 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-etc-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084705 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-env-overrides\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084751 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxt7\" (UniqueName: \"kubernetes.io/projected/3378427f-b215-4a67-8a9d-aab1f8b05d8f-kube-api-access-txxt7\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084833 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e9850e7-1c78-455a-b2d0-77971e830f80-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084860 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaee23aa-3918-44a3-a35c-7a77133e45c7-host-slash\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084873 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-env-overrides\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084883 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-modprobe-d\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084910 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-etc-kubernetes\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084930 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-kubelet\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.084957 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovnkube-config\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085003 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovnkube-script-lib\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085056 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaee23aa-3918-44a3-a35c-7a77133e45c7-iptables-alerter-script\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:59:06.085067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085075 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysctl-conf\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085111 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-cni-multus\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085133 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovnkube-config\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085170 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-log-socket\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085192 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlxl\" (UniqueName: \"kubernetes.io/projected/fe326448-785b-4e7a-a189-76298b83c9b3-kube-api-access-jxlxl\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085208 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085222 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-systemd\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085237 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3378427f-b215-4a67-8a9d-aab1f8b05d8f-ovnkube-script-lib\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085247 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fbdc7ee-c57c-4a65-ba85-b20706762749-host\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085270 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-system-cni-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085317 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085342 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-systemd-units\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085373 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085411 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8d5f4689-033d-4aa8-8603-a56cbdd449cb-kubelet-config\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085437 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysctl-d\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.085450 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085451 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-host\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085485 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46730579-83ec-4a31-8a10-759a76d63eac-tmp\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085515 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-netns\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085564 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-cnibin\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085564 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46730579-83ec-4a31-8a10-759a76d63eac-tmp\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085595 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk9n\" (UniqueName: \"kubernetes.io/projected/8d832341-de73-4a7b-9ce0-c41bbe655035-kube-api-access-mwk9n\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085614 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-sys-fs\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085629 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a76e41e-99b4-4768-a589-014a65e868ac-cert\") pod \"ingress-canary-kkrxc\" (UID: \"1a76e41e-99b4-4768-a589-014a65e868ac\") " pod="openshift-ingress-canary/ingress-canary-kkrxc" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085644 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-kubelet\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085657 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-hosts-file\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085672 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-run-netns\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085687 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysconfig\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085737 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qk6d\" (UniqueName: \"kubernetes.io/projected/9fbdc7ee-c57c-4a65-ba85-b20706762749-kube-api-access-7qk6d\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085820 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-socket-dir-parent\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085853 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-hostroot\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085877 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-ovn\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.085916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085912 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-socket-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085935 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-device-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085950 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-sys\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.085992 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-multus-certs\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086019 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-system-cni-dir\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086058 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086086 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0fc916e0-504f-4aea-85ac-37562156d41e-crio-socket\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086106 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-slash\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086131 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086146 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-cni-netd\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086171 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-tmp-dir\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086187 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zx74\" (UniqueName: \"kubernetes.io/projected/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-kube-api-access-9zx74\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086228 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d399cdee-b235-47b8-96e5-1687506510d0-config-volume\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086272 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086285 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-tmp-dir\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086321 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9tc\" (UniqueName: \"kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc\") pod \"network-check-target-9gjvc\" (UID: \"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3\") " pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086428 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-var-lib-kubelet\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086471 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-cni-binary-copy\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086500 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-systemd\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086535 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46730579-83ec-4a31-8a10-759a76d63eac-etc-tuned\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.086602 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086561 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-daemon-config\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.087309 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086637 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-var-lib-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.087309 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086642 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46730579-83ec-4a31-8a10-759a76d63eac-etc-tuned\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.087309 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086673 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-cni-binary-copy\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.087309 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086670 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-cni-bin\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.087309 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086743 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-run\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.087309 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086761 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fbdc7ee-c57c-4a65-ba85-b20706762749-serviceca\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:59:06.087309 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.086768 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-daemon-config\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.087309 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.087008 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fbdc7ee-c57c-4a65-ba85-b20706762749-serviceca\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:59:06.095164 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.095146 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 06:59:06.095236 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.095215 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kx4sk\"" Nov 03 06:59:06.095890 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.095878 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 06:59:06.096252 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.096237 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 06:59:06.096305 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.096260 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 06:59:06.096305 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.096268 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 06:59:06.096305 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.096280 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 06:59:06.096477 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.096365 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 06:59:06.096477 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.096383 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 06:59:06.096477 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.096386 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 06:59:06.096477 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.096404 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 06:59:06.096477 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.096417 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 06:59:06.096477 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.096443 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 06:59:06.096477 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.096451 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d399cdee-b235-47b8-96e5-1687506510d0-config-volume\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:06.104754 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.104736 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eff68601-5319-4dc8-93c6-16de197bd812-konnectivity-ca\") pod \"konnectivity-agent-nj9qb\" (UID: \"eff68601-5319-4dc8-93c6-16de197bd812\") " pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:59:06.105882 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.105868 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaee23aa-3918-44a3-a35c-7a77133e45c7-iptables-alerter-script\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:59:06.126189 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.126143 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 06:59:06.126541 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.126525 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-kl8t2\"" Nov 03 06:59:06.126541 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.126537 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 06:59:06.126637 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.126586 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-p8ttv\"" Nov 03 06:59:06.126637 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.126612 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7dznd\"" Nov 03 06:59:06.126823 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.126800 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 06:59:06.126904 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.126831 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxfbc\"" Nov 03 06:59:06.130114 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.127467 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lr4sn\"" Nov 03 06:59:06.130114 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.127619 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nd4b5\"" Nov 03 06:59:06.130114 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.127686 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dftjg\"" Nov 03 06:59:06.130114 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.127702 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 06:59:06.130114 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.127901 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 06:59:06.130114 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.127949 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 06:59:06.134862 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.134845 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eff68601-5319-4dc8-93c6-16de197bd812-agent-certs\") pod \"konnectivity-agent-nj9qb\" (UID: \"eff68601-5319-4dc8-93c6-16de197bd812\") " pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:59:06.136119 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.136103 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d5f4689-033d-4aa8-8603-a56cbdd449cb-original-pull-secret\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:59:06.136159 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.136147 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a76e41e-99b4-4768-a589-014a65e868ac-cert\") pod \"ingress-canary-kkrxc\" (UID: \"1a76e41e-99b4-4768-a589-014a65e868ac\") " pod="openshift-ingress-canary/ingress-canary-kkrxc" Nov 03 06:59:06.144386 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.144366 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7gmw\" (UniqueName: \"kubernetes.io/projected/3e9850e7-1c78-455a-b2d0-77971e830f80-kube-api-access-t7gmw\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.144509 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.144492 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bnh\" (UniqueName: \"kubernetes.io/projected/7627e405-9b18-41b2-9fdb-e39eb1ef7e3b-kube-api-access-b2bnh\") pod \"migrator-66df5d496b-2tcnw\" (UID: \"7627e405-9b18-41b2-9fdb-e39eb1ef7e3b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2tcnw" Nov 03 06:59:06.144576 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.144531 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg99r\" (UniqueName: \"kubernetes.io/projected/46730579-83ec-4a31-8a10-759a76d63eac-kube-api-access-wg99r\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.144904 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.144883 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jtj6\" (UniqueName: \"kubernetes.io/projected/d399cdee-b235-47b8-96e5-1687506510d0-kube-api-access-4jtj6\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:06.144955 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.144940 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpzg2\" (UniqueName: \"kubernetes.io/projected/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-kube-api-access-xpzg2\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.145035 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.145020 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjn2\" (UniqueName: \"kubernetes.io/projected/0fc916e0-504f-4aea-85ac-37562156d41e-kube-api-access-9wjn2\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.149745 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.149727 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2z4q\" (UniqueName: \"kubernetes.io/projected/eaee23aa-3918-44a3-a35c-7a77133e45c7-kube-api-access-f2z4q\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:59:06.149955 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.149941 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxt7\" (UniqueName: \"kubernetes.io/projected/3378427f-b215-4a67-8a9d-aab1f8b05d8f-kube-api-access-txxt7\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.150119 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.150100 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwk9n\" (UniqueName: \"kubernetes.io/projected/8d832341-de73-4a7b-9ce0-c41bbe655035-kube-api-access-mwk9n\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:59:06.150199 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.150187 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmkk6\" (UniqueName: \"kubernetes.io/projected/1a76e41e-99b4-4768-a589-014a65e868ac-kube-api-access-lmkk6\") pod \"ingress-canary-kkrxc\" (UID: \"1a76e41e-99b4-4768-a589-014a65e868ac\") " pod="openshift-ingress-canary/ingress-canary-kkrxc" Nov 03 06:59:06.154078 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.154063 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq9tc\" (UniqueName: \"kubernetes.io/projected/f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3-kube-api-access-fq9tc\") pod \"network-check-target-9gjvc\" (UID: \"f1a741d6-8992-4f8f-8cd7-94ea7c3ccff3\") " pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:59:06.154078 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.154071 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qk6d\" (UniqueName: \"kubernetes.io/projected/9fbdc7ee-c57c-4a65-ba85-b20706762749-kube-api-access-7qk6d\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:59:06.154165 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.154074 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zx74\" (UniqueName: \"kubernetes.io/projected/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-kube-api-access-9zx74\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:59:06.155464 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.155444 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlxl\" (UniqueName: \"kubernetes.io/projected/fe326448-785b-4e7a-a189-76298b83c9b3-kube-api-access-jxlxl\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.187776 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187751 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-var-lib-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.187863 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187780 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-cni-bin\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.187863 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187795 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-run\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.187863 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187809 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-cnibin\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.187863 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187824 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.187863 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187839 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-conf-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.187863 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187853 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-cni-bin\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.187863 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187851 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-var-lib-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.187863 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187858 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-cni-bin\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187875 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-lib-modules\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187887 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-cnibin\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187897 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-k8s-cni-cncf-io\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187902 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-cni-bin\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187911 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-conf-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187922 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187930 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-k8s-cni-cncf-io\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187896 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187945 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-node-log\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187960 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-registration-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187896 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-run\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.187994 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8d5f4689-033d-4aa8-8603-a56cbdd449cb-dbus\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188005 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-lib-modules\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188027 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-registration-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188011 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-node-log\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.188033 4409 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188047 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188080 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188097 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-kubernetes\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188115 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188119 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-cni-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188051 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8d5f4689-033d-4aa8-8603-a56cbdd449cb-dbus\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188145 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.188154 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs podName:8d832341-de73-4a7b-9ce0-c41bbe655035 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:06.688133448 +0000 UTC m=+1.880675200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs") pod "network-metrics-daemon-cd7lc" (UID: "8d832341-de73-4a7b-9ce0-c41bbe655035") : secret "metrics-daemon-secret" not found Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188171 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-kubernetes\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188174 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-cni-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188193 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188218 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-etc-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188242 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaee23aa-3918-44a3-a35c-7a77133e45c7-host-slash\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188254 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-etc-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.188242 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188263 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-modprobe-d\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188286 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-etc-kubernetes\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188289 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaee23aa-3918-44a3-a35c-7a77133e45c7-host-slash\") pod \"iptables-alerter-s5f7z\" (UID: \"eaee23aa-3918-44a3-a35c-7a77133e45c7\") " pod="openshift-network-operator/iptables-alerter-s5f7z" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188309 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-kubelet\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188336 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-etc-kubernetes\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188336 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysctl-conf\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188362 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-modprobe-d\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188367 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-cni-multus\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188368 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-kubelet\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188394 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-log-socket\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188403 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-cni-multus\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188423 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-systemd\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188444 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fbdc7ee-c57c-4a65-ba85-b20706762749-host\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188453 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysctl-conf\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188464 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-log-socket\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188496 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fbdc7ee-c57c-4a65-ba85-b20706762749-host\") pod \"node-ca-nn9v4\" (UID: \"9fbdc7ee-c57c-4a65-ba85-b20706762749\") " pod="openshift-image-registry/node-ca-nn9v4" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188465 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-system-cni-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188508 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-system-cni-dir\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188519 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188538 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-systemd-units\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188563 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188571 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-systemd-units\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188509 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-systemd\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.188588 4409 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188617 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188625 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8d5f4689-033d-4aa8-8603-a56cbdd449cb-kubelet-config\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188587 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8d5f4689-033d-4aa8-8603-a56cbdd449cb-kubelet-config\") pod \"global-pull-secret-syncer-zhpcz\" (UID: \"8d5f4689-033d-4aa8-8603-a56cbdd449cb\") " pod="kube-system/global-pull-secret-syncer-zhpcz" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.188637 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls podName:0fc916e0-504f-4aea-85ac-37562156d41e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:06.688621862 +0000 UTC m=+1.881163615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqbfp" (UID: "0fc916e0-504f-4aea-85ac-37562156d41e") : secret "insights-runtime-extractor-tls" not found Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188662 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysctl-d\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188675 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-host\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188689 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-netns\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188703 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-cnibin\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188733 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-sys-fs\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188744 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-host\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188758 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-kubelet\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188764 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-netns\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188779 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-hosts-file\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188789 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-sys-fs\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188800 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-run-netns\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188819 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-var-lib-kubelet\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188823 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysconfig\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188790 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-cnibin\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188832 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysctl-d\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188832 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fa0cc98-1f99-4197-9a87-52c9ea4277f9-hosts-file\") pod \"node-resolver-mwws4\" (UID: \"2fa0cc98-1f99-4197-9a87-52c9ea4277f9\") " pod="openshift-dns/node-resolver-mwws4" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188848 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fc916e0-504f-4aea-85ac-37562156d41e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188846 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-run-netns\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188847 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-socket-dir-parent\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188891 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-multus-socket-dir-parent\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188893 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-hostroot\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188913 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-etc-sysconfig\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188918 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-ovn\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188946 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-socket-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188951 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-ovn\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188970 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-device-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188986 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-hostroot\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188993 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-sys\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.188990 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-socket-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189012 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-device-dir\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189015 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-multus-certs\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189038 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6551e74b-f4c1-4be3-8fbf-cf5f262caea4-host-run-multus-certs\") pod \"multus-rww5p\" (UID: \"6551e74b-f4c1-4be3-8fbf-cf5f262caea4\") " pod="openshift-multus/multus-rww5p" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189039 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-system-cni-dir\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189040 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-sys\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189060 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189074 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0fc916e0-504f-4aea-85ac-37562156d41e-crio-socket\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189087 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-slash\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189101 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189123 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-cni-netd\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189144 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e9850e7-1c78-455a-b2d0-77971e830f80-system-cni-dir\") pod \"multus-additional-cni-plugins-4ndld\" (UID: \"3e9850e7-1c78-455a-b2d0-77971e830f80\") " pod="openshift-multus/multus-additional-cni-plugins-4ndld" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189152 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-slash\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189168 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-host-cni-netd\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.189169 4409 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189183 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0fc916e0-504f-4aea-85ac-37562156d41e-crio-socket\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189185 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189210 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-var-lib-kubelet\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189210 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe326448-785b-4e7a-a189-76298b83c9b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-22n8v\" (UID: \"fe326448-785b-4e7a-a189-76298b83c9b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22n8v" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.189221 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls podName:d399cdee-b235-47b8-96e5-1687506510d0 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:06.689207925 +0000 UTC m=+1.881749654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls") pod "dns-default-htqcm" (UID: "d399cdee-b235-47b8-96e5-1687506510d0") : secret "dns-default-metrics-tls" not found Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189160 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-openvswitch\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189242 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46730579-83ec-4a31-8a10-759a76d63eac-var-lib-kubelet\") pod \"tuned-6f246\" (UID: \"46730579-83ec-4a31-8a10-759a76d63eac\") " pod="openshift-cluster-node-tuning-operator/tuned-6f246" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189246 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-systemd\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.189383 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.189264 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3378427f-b215-4a67-8a9d-aab1f8b05d8f-run-systemd\") pod \"ovnkube-node-l4ffc\" (UID: \"3378427f-b215-4a67-8a9d-aab1f8b05d8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:06.692483 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.692447 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:06.692483 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.692487 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:06.692666 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:06.692528 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:59:06.692666 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.692598 4409 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 06:59:06.692666 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.692597 4409 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:59:06.692666 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.692660 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs podName:8d832341-de73-4a7b-9ce0-c41bbe655035 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:07.692644402 +0000 UTC m=+2.885186144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs") pod "network-metrics-daemon-cd7lc" (UID: "8d832341-de73-4a7b-9ce0-c41bbe655035") : secret "metrics-daemon-secret" not found Nov 03 06:59:06.692810 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.692671 4409 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:59:06.692810 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.692687 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls podName:0fc916e0-504f-4aea-85ac-37562156d41e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:07.692674781 +0000 UTC m=+2.885216516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqbfp" (UID: "0fc916e0-504f-4aea-85ac-37562156d41e") : secret "insights-runtime-extractor-tls" not found Nov 03 06:59:06.692810 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:06.692763 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls podName:d399cdee-b235-47b8-96e5-1687506510d0 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:07.692749229 +0000 UTC m=+2.885290957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls") pod "dns-default-htqcm" (UID: "d399cdee-b235-47b8-96e5-1687506510d0") : secret "dns-default-metrics-tls" not found Nov 03 06:59:07.228273 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:07.228245 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mwws4_2fa0cc98-1f99-4197-9a87-52c9ea4277f9/dns-node-resolver/0.log" Nov 03 06:59:07.698108 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:07.698078 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:07.698279 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:07.698120 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:59:07.698279 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:07.698146 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:07.698279 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:07.698228 4409 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 06:59:07.698279 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:07.698238 4409 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:59:07.698279 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:07.698242 4409 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 06:59:07.698462 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:07.698289 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls podName:d399cdee-b235-47b8-96e5-1687506510d0 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.698274814 +0000 UTC m=+4.890816546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls") pod "dns-default-htqcm" (UID: "d399cdee-b235-47b8-96e5-1687506510d0") : secret "dns-default-metrics-tls" not found Nov 03 06:59:07.698462 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:07.698304 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls podName:0fc916e0-504f-4aea-85ac-37562156d41e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.698297996 +0000 UTC m=+4.890839725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqbfp" (UID: "0fc916e0-504f-4aea-85ac-37562156d41e") : secret "insights-runtime-extractor-tls" not found Nov 03 06:59:07.698462 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:07.698316 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs podName:8d832341-de73-4a7b-9ce0-c41bbe655035 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:09.698310891 +0000 UTC m=+4.890852619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs") pod "network-metrics-daemon-cd7lc" (UID: "8d832341-de73-4a7b-9ce0-c41bbe655035") : secret "metrics-daemon-secret" not found Nov 03 06:59:08.227908 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:08.227882 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nn9v4_9fbdc7ee-c57c-4a65-ba85-b20706762749/node-ca/0.log" Nov 03 06:59:08.937220 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:08.937193 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:09.227672 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:09.227601 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kkrxc_1a76e41e-99b4-4768-a589-014a65e868ac/serve-healthcheck-canary/0.log" Nov 03 06:59:09.473665 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:09.473635 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:59:09.628411 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:09.628384 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-2tcnw_7627e405-9b18-41b2-9fdb-e39eb1ef7e3b/migrator/0.log" Nov 03 06:59:09.712277 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:09.712256 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:09.712397 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:09.712298 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:09.712397 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:09.712330 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:59:09.712397 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:09.712386 4409 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 06:59:09.712562 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:09.712410 4409 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 06:59:09.712562 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:09.712453 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls podName:0fc916e0-504f-4aea-85ac-37562156d41e nodeName:}" failed. No retries permitted until 2026-11-03 06:59:13.712439637 +0000 UTC m=+8.904981365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqbfp" (UID: "0fc916e0-504f-4aea-85ac-37562156d41e") : secret "insights-runtime-extractor-tls" not found Nov 03 06:59:09.712562 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:09.712468 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs podName:8d832341-de73-4a7b-9ce0-c41bbe655035 nodeName:}" failed. No retries permitted until 2026-11-03 06:59:13.712461637 +0000 UTC m=+8.905003365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs") pod "network-metrics-daemon-cd7lc" (UID: "8d832341-de73-4a7b-9ce0-c41bbe655035") : secret "metrics-daemon-secret" not found Nov 03 06:59:09.712785 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:09.712763 4409 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 06:59:09.715113 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:09.715089 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d399cdee-b235-47b8-96e5-1687506510d0-metrics-tls\") pod \"dns-default-htqcm\" (UID: \"d399cdee-b235-47b8-96e5-1687506510d0\") " pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:09.927327 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:09.927255 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:10.312792 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:10.312763 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-htqcm"] Nov 03 06:59:10.316558 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:10.316532 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd399cdee_b235_47b8_96e5_1687506510d0.slice/crio-1d60dd3289d4f065c5784c3c08f3d94235f98c8ee73240f9d50e7e0bc398f303 WatchSource:0}: Error finding container 1d60dd3289d4f065c5784c3c08f3d94235f98c8ee73240f9d50e7e0bc398f303: Status 404 returned error can't find the container with id 1d60dd3289d4f065c5784c3c08f3d94235f98c8ee73240f9d50e7e0bc398f303 Nov 03 06:59:10.318645 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:10.318629 4409 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 06:59:10.956360 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:10.956321 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-htqcm" event={"ID":"d399cdee-b235-47b8-96e5-1687506510d0","Type":"ContainerStarted","Data":"1d60dd3289d4f065c5784c3c08f3d94235f98c8ee73240f9d50e7e0bc398f303"} Nov 03 06:59:11.136353 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:11.136325 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:11.136609 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:11.136591 4409 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 06:59:12.963853 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:12.963817 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-htqcm" event={"ID":"d399cdee-b235-47b8-96e5-1687506510d0","Type":"ContainerStarted","Data":"f539e23cd1bae3eb5c0be8485d4b89361c1611de411ea8d8d1ea3fa6164eea21"} Nov 03 06:59:12.964250 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:12.963877 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-htqcm" event={"ID":"d399cdee-b235-47b8-96e5-1687506510d0","Type":"ContainerStarted","Data":"cada7bd04119132aa03a0ebbbcabb1b851391bf0cec3775dce4fd7b0e69d5af3"} Nov 03 06:59:12.964250 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:12.963956 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:13.736601 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:13.736561 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:59:13.736811 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:13.736609 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:13.739151 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:13.739122 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fc916e0-504f-4aea-85ac-37562156d41e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqbfp\" (UID: \"0fc916e0-504f-4aea-85ac-37562156d41e\") " pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:13.739291 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:13.739122 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d832341-de73-4a7b-9ce0-c41bbe655035-metrics-certs\") pod \"network-metrics-daemon-cd7lc\" (UID: \"8d832341-de73-4a7b-9ce0-c41bbe655035\") " pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:59:13.826215 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:13.826191 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd7lc" Nov 03 06:59:13.827294 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:13.827266 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqbfp" Nov 03 06:59:14.217221 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:14.217187 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cd7lc"] Nov 03 06:59:14.237335 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:14.237314 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d832341_de73_4a7b_9ce0_c41bbe655035.slice/crio-fe8205b9ee8b2c2453133565b9bad1f3bdf29f7c1ce107f32c206e0d8853e379 WatchSource:0}: Error finding container fe8205b9ee8b2c2453133565b9bad1f3bdf29f7c1ce107f32c206e0d8853e379: Status 404 returned error can't find the container with id fe8205b9ee8b2c2453133565b9bad1f3bdf29f7c1ce107f32c206e0d8853e379 Nov 03 06:59:14.249659 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:14.249634 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqbfp"] Nov 03 06:59:14.253332 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:14.253296 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc916e0_504f_4aea_85ac_37562156d41e.slice/crio-19cd6f8ffa30aa9dd5fad92f375396b543143ce665ae3bfee12e88ea321be543 WatchSource:0}: Error finding container 19cd6f8ffa30aa9dd5fad92f375396b543143ce665ae3bfee12e88ea321be543: Status 404 returned error can't find the container with id 19cd6f8ffa30aa9dd5fad92f375396b543143ce665ae3bfee12e88ea321be543 Nov 03 06:59:14.976800 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:14.976764 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqbfp" event={"ID":"0fc916e0-504f-4aea-85ac-37562156d41e","Type":"ContainerStarted","Data":"6652cc0acdbabf9d5744c2108e5aadee365d1feeaa80942d4aed7fb0bffa2b1c"} Nov 03 06:59:14.976982 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:14.976849 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqbfp" event={"ID":"0fc916e0-504f-4aea-85ac-37562156d41e","Type":"ContainerStarted","Data":"19cd6f8ffa30aa9dd5fad92f375396b543143ce665ae3bfee12e88ea321be543"} Nov 03 06:59:14.978007 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:14.977976 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cd7lc" event={"ID":"8d832341-de73-4a7b-9ce0-c41bbe655035","Type":"ContainerStarted","Data":"fe8205b9ee8b2c2453133565b9bad1f3bdf29f7c1ce107f32c206e0d8853e379"} Nov 03 06:59:15.449614 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:15.449586 4409 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-121.us-west-1.compute.internal" event="NodeReady" Nov 03 06:59:15.982762 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:15.982669 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqbfp" event={"ID":"0fc916e0-504f-4aea-85ac-37562156d41e","Type":"ContainerStarted","Data":"d2b14bf300980ba88d2c40f6715101f8216adf5041de5dd9a9fa3607a260909f"} Nov 03 06:59:15.984378 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:15.984355 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cd7lc" event={"ID":"8d832341-de73-4a7b-9ce0-c41bbe655035","Type":"ContainerStarted","Data":"d138d65bc04bad6adeefc5fd5fe6e11a90b4a766e632157f7ff55580af5caad5"} Nov 03 06:59:15.984480 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:15.984397 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cd7lc" event={"ID":"8d832341-de73-4a7b-9ce0-c41bbe655035","Type":"ContainerStarted","Data":"6e952135e9c3b7e75f1d781537358a6e8684da9333da494bbe863a8570daaf3a"} Nov 03 06:59:17.992471 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:17.992434 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqbfp" event={"ID":"0fc916e0-504f-4aea-85ac-37562156d41e","Type":"ContainerStarted","Data":"432c681db92624f46341b7324dfa4ddc8203de0cac3bb9f98b0221d0caed2612"} Nov 03 06:59:22.969421 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:22.969391 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-htqcm" Nov 03 06:59:26.049013 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.048979 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-mchvh"] Nov 03 06:59:26.050943 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.050927 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.120073 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.120041 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82cb9f11-c15b-4ae1-885a-9474727d453b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mchvh\" (UID: \"82cb9f11-c15b-4ae1-885a-9474727d453b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.120200 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.120079 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82cb9f11-c15b-4ae1-885a-9474727d453b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mchvh\" (UID: \"82cb9f11-c15b-4ae1-885a-9474727d453b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.120200 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.120153 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82cb9f11-c15b-4ae1-885a-9474727d453b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mchvh\" (UID: \"82cb9f11-c15b-4ae1-885a-9474727d453b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.120200 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.120192 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh576\" (UniqueName: \"kubernetes.io/projected/82cb9f11-c15b-4ae1-885a-9474727d453b-kube-api-access-sh576\") pod \"prometheus-operator-9cd89b74c-mchvh\" (UID: \"82cb9f11-c15b-4ae1-885a-9474727d453b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.153952 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.153928 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 06:59:26.155249 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.155235 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 06:59:26.155298 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.155279 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 06:59:26.155340 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.155313 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 06:59:26.155379 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.155363 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 06:59:26.203415 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.203393 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lvz8f\"" Nov 03 06:59:26.204140 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.204120 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-mchvh"] Nov 03 06:59:26.220961 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.220941 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82cb9f11-c15b-4ae1-885a-9474727d453b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mchvh\" (UID: \"82cb9f11-c15b-4ae1-885a-9474727d453b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.221023 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.220965 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82cb9f11-c15b-4ae1-885a-9474727d453b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mchvh\" (UID: \"82cb9f11-c15b-4ae1-885a-9474727d453b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.221023 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.220985 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82cb9f11-c15b-4ae1-885a-9474727d453b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mchvh\" (UID: \"82cb9f11-c15b-4ae1-885a-9474727d453b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.221023 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.221000 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh576\" (UniqueName: \"kubernetes.io/projected/82cb9f11-c15b-4ae1-885a-9474727d453b-kube-api-access-sh576\") pod \"prometheus-operator-9cd89b74c-mchvh\" (UID: \"82cb9f11-c15b-4ae1-885a-9474727d453b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.221623 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.221601 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82cb9f11-c15b-4ae1-885a-9474727d453b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-mchvh\" (UID: \"82cb9f11-c15b-4ae1-885a-9474727d453b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.223448 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.223432 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82cb9f11-c15b-4ae1-885a-9474727d453b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-mchvh\" (UID: \"82cb9f11-c15b-4ae1-885a-9474727d453b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.223544 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.223526 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82cb9f11-c15b-4ae1-885a-9474727d453b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-mchvh\" (UID: \"82cb9f11-c15b-4ae1-885a-9474727d453b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.284047 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.284024 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh576\" (UniqueName: \"kubernetes.io/projected/82cb9f11-c15b-4ae1-885a-9474727d453b-kube-api-access-sh576\") pod \"prometheus-operator-9cd89b74c-mchvh\" (UID: \"82cb9f11-c15b-4ae1-885a-9474727d453b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.361059 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.361037 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" Nov 03 06:59:26.699392 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.699319 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57999f5f57-f8mz2"] Nov 03 06:59:26.701981 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.701962 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.733337 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.733315 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-mchvh"] Nov 03 06:59:26.736098 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:26.736071 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82cb9f11_c15b_4ae1_885a_9474727d453b.slice/crio-ed64f0432a2a33bc45445cc1a7f2f207302350a3010916fe0c5d17a99ed66956 WatchSource:0}: Error finding container ed64f0432a2a33bc45445cc1a7f2f207302350a3010916fe0c5d17a99ed66956: Status 404 returned error can't find the container with id ed64f0432a2a33bc45445cc1a7f2f207302350a3010916fe0c5d17a99ed66956 Nov 03 06:59:26.756685 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.756664 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 06:59:26.756879 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.756863 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 06:59:26.756959 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.756876 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 06:59:26.756959 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.756906 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 06:59:26.782173 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.782152 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 06:59:26.782267 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.782252 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h5pkt\"" Nov 03 06:59:26.782316 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.782279 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 06:59:26.782316 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.782296 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 06:59:26.782935 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.782917 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57999f5f57-f8mz2"] Nov 03 06:59:26.824529 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.824507 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-oauth-serving-cert\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.824631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.824537 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fnm6\" (UniqueName: \"kubernetes.io/projected/40548b01-8d84-4461-b36a-92fdafcb75c5-kube-api-access-9fnm6\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.824631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.824555 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-service-ca\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.824631 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.824578 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-oauth-config\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.824795 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.824651 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-serving-cert\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.824795 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.824690 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-console-config\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.824865 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.824815 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40548b01-8d84-4461-b36a-92fdafcb75c5-tmp\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.925318 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.925298 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-oauth-config\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.925429 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.925330 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-serving-cert\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.925429 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.925347 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-console-config\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.925429 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.925374 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40548b01-8d84-4461-b36a-92fdafcb75c5-tmp\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.925587 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.925508 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-oauth-serving-cert\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.925587 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.925562 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fnm6\" (UniqueName: \"kubernetes.io/projected/40548b01-8d84-4461-b36a-92fdafcb75c5-kube-api-access-9fnm6\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.925685 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.925600 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-service-ca\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.925799 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.925779 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40548b01-8d84-4461-b36a-92fdafcb75c5-tmp\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.926167 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.926144 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-console-config\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.926274 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.926169 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-oauth-serving-cert\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.926335 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.926296 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-service-ca\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.927882 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.927858 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-oauth-config\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.927988 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.927973 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-serving-cert\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:26.988583 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:26.988536 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fnm6\" (UniqueName: \"kubernetes.io/projected/40548b01-8d84-4461-b36a-92fdafcb75c5-kube-api-access-9fnm6\") pod \"console-57999f5f57-f8mz2\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:27.015489 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:27.015471 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:27.020804 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:27.020757 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" event={"ID":"82cb9f11-c15b-4ae1-885a-9474727d453b","Type":"ContainerStarted","Data":"ed64f0432a2a33bc45445cc1a7f2f207302350a3010916fe0c5d17a99ed66956"} Nov 03 06:59:27.387010 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:27.386977 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57999f5f57-f8mz2"] Nov 03 06:59:27.390301 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:27.390273 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40548b01_8d84_4461_b36a_92fdafcb75c5.slice/crio-a4e01616e59df23b01da0c704a1666b3c15ab1cd0ad4e5b3bf6abc605bde287b WatchSource:0}: Error finding container a4e01616e59df23b01da0c704a1666b3c15ab1cd0ad4e5b3bf6abc605bde287b: Status 404 returned error can't find the container with id a4e01616e59df23b01da0c704a1666b3c15ab1cd0ad4e5b3bf6abc605bde287b Nov 03 06:59:28.025946 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:28.025908 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57999f5f57-f8mz2" event={"ID":"40548b01-8d84-4461-b36a-92fdafcb75c5","Type":"ContainerStarted","Data":"a4e01616e59df23b01da0c704a1666b3c15ab1cd0ad4e5b3bf6abc605bde287b"} Nov 03 06:59:29.031486 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:29.031448 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" event={"ID":"82cb9f11-c15b-4ae1-885a-9474727d453b","Type":"ContainerStarted","Data":"a8e64d7630c64ee71d63a832d728210ab3604ee599d418f28cc3840f559eb526"} Nov 03 06:59:29.031924 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:29.031512 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" event={"ID":"82cb9f11-c15b-4ae1-885a-9474727d453b","Type":"ContainerStarted","Data":"28bc83376f32e7d2161f06141cf7022c521733809b05f5e9e619daf30523c351"} Nov 03 06:59:29.150457 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:29.150120 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-mchvh" podStartSLOduration=1.478796316 podStartE2EDuration="3.150104741s" totalImagesPullingTime="1.671308425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:29.150078403 +0000 UTC m=+24.342620155" watchObservedRunningTime="2026-11-03 06:59:29.150104741 +0000 UTC m=+24.342646482" Nov 03 06:59:30.745731 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:30.745653 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4ffc" Nov 03 06:59:31.044655 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:31.044580 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57999f5f57-f8mz2" event={"ID":"40548b01-8d84-4461-b36a-92fdafcb75c5","Type":"ContainerStarted","Data":"474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657"} Nov 03 06:59:31.164410 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:31.164362 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57999f5f57-f8mz2" podStartSLOduration=2.084913247 podStartE2EDuration="5.164347048s" totalImagesPullingTime="3.079433801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:31.163871803 +0000 UTC m=+26.356413551" watchObservedRunningTime="2026-11-03 06:59:31.164347048 +0000 UTC m=+26.356888798" Nov 03 06:59:31.601949 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:31.601904 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nj9qb" Nov 03 06:59:32.667302 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.667263 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9"] Nov 03 06:59:32.671701 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.670419 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.763116 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.763088 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtr8\" (UniqueName: \"kubernetes.io/projected/8628283a-b865-41bb-9126-b6bb140650d3-kube-api-access-tvtr8\") pod \"openshift-state-metrics-7bd7489957-ms6g9\" (UID: \"8628283a-b865-41bb-9126-b6bb140650d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.763256 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.763123 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8628283a-b865-41bb-9126-b6bb140650d3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ms6g9\" (UID: \"8628283a-b865-41bb-9126-b6bb140650d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.763256 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.763146 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8628283a-b865-41bb-9126-b6bb140650d3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ms6g9\" (UID: \"8628283a-b865-41bb-9126-b6bb140650d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.763256 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.763213 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8628283a-b865-41bb-9126-b6bb140650d3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ms6g9\" (UID: \"8628283a-b865-41bb-9126-b6bb140650d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.773633 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.773614 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 06:59:32.773917 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.773898 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-p5s9d\"" Nov 03 06:59:32.774018 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.773919 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 06:59:32.774597 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.774581 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9"] Nov 03 06:59:32.777542 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.777528 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s92rs"] Nov 03 06:59:32.779319 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.779305 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.832662 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.832636 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 06:59:32.832662 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.832645 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 06:59:32.832823 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.832673 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-666h2\"" Nov 03 06:59:32.864067 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864048 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8628283a-b865-41bb-9126-b6bb140650d3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ms6g9\" (UID: \"8628283a-b865-41bb-9126-b6bb140650d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.864175 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864074 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-textfile\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.864175 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864092 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-accelerators-collector-config\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.864175 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864108 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-wtmp\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.864175 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864123 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8628283a-b865-41bb-9126-b6bb140650d3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ms6g9\" (UID: \"8628283a-b865-41bb-9126-b6bb140650d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.864347 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864326 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d325d30-1335-4213-9d57-d86d25726e89-metrics-client-ca\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.864402 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864354 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.864447 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864418 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vktqk\" (UniqueName: \"kubernetes.io/projected/8d325d30-1335-4213-9d57-d86d25726e89-kube-api-access-vktqk\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.864481 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864466 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d325d30-1335-4213-9d57-d86d25726e89-root\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.864529 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864496 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d325d30-1335-4213-9d57-d86d25726e89-sys\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.864580 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864551 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-tls\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.864637 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864605 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtr8\" (UniqueName: \"kubernetes.io/projected/8628283a-b865-41bb-9126-b6bb140650d3-kube-api-access-tvtr8\") pod \"openshift-state-metrics-7bd7489957-ms6g9\" (UID: \"8628283a-b865-41bb-9126-b6bb140650d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.864690 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864647 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8628283a-b865-41bb-9126-b6bb140650d3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ms6g9\" (UID: \"8628283a-b865-41bb-9126-b6bb140650d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.864809 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.864790 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8628283a-b865-41bb-9126-b6bb140650d3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-ms6g9\" (UID: \"8628283a-b865-41bb-9126-b6bb140650d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.866498 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.866479 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8628283a-b865-41bb-9126-b6bb140650d3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-ms6g9\" (UID: \"8628283a-b865-41bb-9126-b6bb140650d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.866963 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.866943 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8628283a-b865-41bb-9126-b6bb140650d3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-ms6g9\" (UID: \"8628283a-b865-41bb-9126-b6bb140650d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.917699 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.917653 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 06:59:32.927890 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.927869 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtr8\" (UniqueName: \"kubernetes.io/projected/8628283a-b865-41bb-9126-b6bb140650d3-kube-api-access-tvtr8\") pod \"openshift-state-metrics-7bd7489957-ms6g9\" (UID: \"8628283a-b865-41bb-9126-b6bb140650d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:32.965889 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.965867 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d325d30-1335-4213-9d57-d86d25726e89-sys\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.965981 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.965894 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-tls\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.965981 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.965920 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-textfile\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.965981 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.965935 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-accelerators-collector-config\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.965981 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.965951 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-wtmp\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.965981 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.965971 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d325d30-1335-4213-9d57-d86d25726e89-metrics-client-ca\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.966225 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.965982 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d325d30-1335-4213-9d57-d86d25726e89-sys\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.966225 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.966097 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.966225 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.966112 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-wtmp\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.966225 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.966137 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vktqk\" (UniqueName: \"kubernetes.io/projected/8d325d30-1335-4213-9d57-d86d25726e89-kube-api-access-vktqk\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.966225 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.966171 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d325d30-1335-4213-9d57-d86d25726e89-root\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.966473 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.966249 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d325d30-1335-4213-9d57-d86d25726e89-root\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.966473 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.966316 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-textfile\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.966766 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.966581 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d325d30-1335-4213-9d57-d86d25726e89-metrics-client-ca\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.966766 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.966602 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-accelerators-collector-config\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.968212 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.968194 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-tls\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.968334 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.968318 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d325d30-1335-4213-9d57-d86d25726e89-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:32.985284 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:32.985263 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" Nov 03 06:59:33.030078 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:33.030054 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vktqk\" (UniqueName: \"kubernetes.io/projected/8d325d30-1335-4213-9d57-d86d25726e89-kube-api-access-vktqk\") pod \"node-exporter-s92rs\" (UID: \"8d325d30-1335-4213-9d57-d86d25726e89\") " pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:33.090288 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:33.090262 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s92rs" Nov 03 06:59:33.099287 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:33.099263 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d325d30_1335_4213_9d57_d86d25726e89.slice/crio-e9834a3240e13f2d5f2e2d2b856441d25577fd00946bc9fcad460410ed752542 WatchSource:0}: Error finding container e9834a3240e13f2d5f2e2d2b856441d25577fd00946bc9fcad460410ed752542: Status 404 returned error can't find the container with id e9834a3240e13f2d5f2e2d2b856441d25577fd00946bc9fcad460410ed752542 Nov 03 06:59:33.403246 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:33.403210 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9"] Nov 03 06:59:33.405962 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:33.405936 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8628283a_b865_41bb_9126_b6bb140650d3.slice/crio-96546331747eb1e016505351aa18b0e3b4f477bd1f1b4de7f83ab2ccf8438bd9 WatchSource:0}: Error finding container 96546331747eb1e016505351aa18b0e3b4f477bd1f1b4de7f83ab2ccf8438bd9: Status 404 returned error can't find the container with id 96546331747eb1e016505351aa18b0e3b4f477bd1f1b4de7f83ab2ccf8438bd9 Nov 03 06:59:34.058386 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:34.058323 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s92rs" event={"ID":"8d325d30-1335-4213-9d57-d86d25726e89","Type":"ContainerStarted","Data":"e9834a3240e13f2d5f2e2d2b856441d25577fd00946bc9fcad460410ed752542"} Nov 03 06:59:34.060889 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:34.060856 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" event={"ID":"8628283a-b865-41bb-9126-b6bb140650d3","Type":"ContainerStarted","Data":"bbb1d9cfafb3051760146c1a755d278a215ff6822a750ff0a6c28fa2ac4b0a0e"} Nov 03 06:59:34.061028 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:34.060915 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" event={"ID":"8628283a-b865-41bb-9126-b6bb140650d3","Type":"ContainerStarted","Data":"c812556f6ae762457b66b4680e2114e1dded867e604657ebd8cc332d814e6934"} Nov 03 06:59:34.061028 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:34.060957 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" event={"ID":"8628283a-b865-41bb-9126-b6bb140650d3","Type":"ContainerStarted","Data":"96546331747eb1e016505351aa18b0e3b4f477bd1f1b4de7f83ab2ccf8438bd9"} Nov 03 06:59:35.065510 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.065476 4409 generic.go:356] "Generic (PLEG): container finished" podID="8d325d30-1335-4213-9d57-d86d25726e89" containerID="0c5e5b9f31f53db3f76a2ef84db100aa7cf69cc611e9b97189ee9c9ced5329e3" exitCode=0 Nov 03 06:59:35.065829 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.065558 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s92rs" event={"ID":"8d325d30-1335-4213-9d57-d86d25726e89","Type":"ContainerDied","Data":"0c5e5b9f31f53db3f76a2ef84db100aa7cf69cc611e9b97189ee9c9ced5329e3"} Nov 03 06:59:35.169210 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.169107 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 06:59:35.191030 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.191009 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.283916 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.283887 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.284057 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.283922 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.284057 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.283944 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.284057 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.283970 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.284057 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.284047 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.284185 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.284073 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.284185 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.284100 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.284185 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.284116 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.284185 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.284164 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.284293 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.284265 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-out\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.284325 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.284288 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.284384 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.284369 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-web-config\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.284415 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.284401 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87lc7\" (UniqueName: \"kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-kube-api-access-87lc7\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.297952 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.297926 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 06:59:35.299260 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.299238 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 06:59:35.341116 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.341051 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 06:59:35.341116 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.341100 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 06:59:35.341273 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.341104 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 06:59:35.341273 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.341108 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 06:59:35.341273 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.341136 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 06:59:35.341273 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.341131 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 06:59:35.341273 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.341143 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 06:59:35.341273 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.341210 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 06:59:35.341273 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.341261 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tjb9h\"" Nov 03 06:59:35.385106 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385081 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385206 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385109 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385206 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385134 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385206 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385150 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385206 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385176 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385400 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385301 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-out\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385400 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385353 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385400 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385395 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-web-config\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385546 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385441 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87lc7\" (UniqueName: \"kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-kube-api-access-87lc7\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385546 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385473 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385546 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385500 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385546 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385534 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385757 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385560 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.385947 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385917 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.386038 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.385947 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.386996 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.386846 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.388493 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.388464 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.389041 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.388994 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.389261 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.389236 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-out\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.389442 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.389414 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.389530 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.389419 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.389594 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.389575 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.389643 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.389624 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.390066 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.390050 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.390194 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.390178 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-web-config\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.448207 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.448187 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87lc7\" (UniqueName: \"kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-kube-api-access-87lc7\") pod \"alertmanager-main-0\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:35.632298 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:35.632234 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 06:59:36.026165 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.026072 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c495b4dd7-thgpm"] Nov 03 06:59:36.054897 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.054871 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.085671 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.085645 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" event={"ID":"8628283a-b865-41bb-9126-b6bb140650d3","Type":"ContainerStarted","Data":"ecdc350be664430bca1aa2518302189a8d82492b1f527325c7eb2f6e093e1db5"} Nov 03 06:59:36.087700 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.087676 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s92rs" event={"ID":"8d325d30-1335-4213-9d57-d86d25726e89","Type":"ContainerStarted","Data":"8eded65e0d0a1f7b5e8cc444f2d603aa71bca61a2d29d5173fad79401b47d0f6"} Nov 03 06:59:36.087822 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.087738 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s92rs" event={"ID":"8d325d30-1335-4213-9d57-d86d25726e89","Type":"ContainerStarted","Data":"1d1ec3a409595c4856c5ebf0c3ce42ef8de85e876bee30b963b7dcc7767a8862"} Nov 03 06:59:36.091459 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.091438 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c495b4dd7-thgpm"] Nov 03 06:59:36.162158 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:36.162136 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4f27a2_ff03_46a0_b793_9210ac6f4e7e.slice/crio-7964052b8e0bb8e47060806f6250e5f9681428215e9b49a157a48edb110ea28e WatchSource:0}: Error finding container 7964052b8e0bb8e47060806f6250e5f9681428215e9b49a157a48edb110ea28e: Status 404 returned error can't find the container with id 7964052b8e0bb8e47060806f6250e5f9681428215e9b49a157a48edb110ea28e Nov 03 06:59:36.164331 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.164304 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 06:59:36.165942 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.165923 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 06:59:36.190655 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.190635 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-oauth-serving-cert\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.190776 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.190684 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb8qw\" (UniqueName: \"kubernetes.io/projected/84852c8e-a22b-40c8-b495-dec0174ef5d1-kube-api-access-jb8qw\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.190776 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.190752 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-config\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.190877 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.190798 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-service-ca\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.191122 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.191098 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-serving-cert\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.191189 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.191154 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-oauth-config\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.191229 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.191197 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84852c8e-a22b-40c8-b495-dec0174ef5d1-tmp\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.191292 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.191274 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-trusted-ca-bundle\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.291828 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.291769 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-serving-cert\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.291828 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.291805 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-oauth-config\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.291988 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.291827 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84852c8e-a22b-40c8-b495-dec0174ef5d1-tmp\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.291988 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.291854 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-trusted-ca-bundle\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.291988 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.291883 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-oauth-serving-cert\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.291988 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.291926 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb8qw\" (UniqueName: \"kubernetes.io/projected/84852c8e-a22b-40c8-b495-dec0174ef5d1-kube-api-access-jb8qw\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.291988 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.291967 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-config\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.292214 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.291991 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-service-ca\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.292814 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.292278 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84852c8e-a22b-40c8-b495-dec0174ef5d1-tmp\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.292814 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.292764 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-service-ca\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.292814 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.292804 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-trusted-ca-bundle\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.292995 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.292920 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-config\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.292995 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.292924 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-oauth-serving-cert\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.294326 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.294304 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-serving-cert\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.294414 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.294344 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-oauth-config\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.309976 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.309853 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-ms6g9" podStartSLOduration=2.424149103 podStartE2EDuration="4.309832783s" totalImagesPullingTime="1.88568368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:36.309824615 +0000 UTC m=+31.502366366" watchObservedRunningTime="2026-11-03 06:59:36.309832783 +0000 UTC m=+31.502374533" Nov 03 06:59:36.355289 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.355268 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb8qw\" (UniqueName: \"kubernetes.io/projected/84852c8e-a22b-40c8-b495-dec0174ef5d1-kube-api-access-jb8qw\") pod \"console-6c495b4dd7-thgpm\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.385096 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.385068 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:36.436550 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.436504 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s92rs" podStartSLOduration=2.6491377959999998 podStartE2EDuration="4.436490969s" totalImagesPullingTime="1.694752846s" totalInitContainerRuntime="92.600327ms" isStatefulPod=true podCreationTimestamp="2026-11-03 06:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:36.43563062 +0000 UTC m=+31.628172368" watchObservedRunningTime="2026-11-03 06:59:36.436490969 +0000 UTC m=+31.629032719" Nov 03 06:59:36.791857 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:36.791823 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c495b4dd7-thgpm"] Nov 03 06:59:36.795164 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:36.795135 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84852c8e_a22b_40c8_b495_dec0174ef5d1.slice/crio-e6b01f51029511d9357217ea3159791eacb6e142be46acb95b2d66431ea4de7e WatchSource:0}: Error finding container e6b01f51029511d9357217ea3159791eacb6e142be46acb95b2d66431ea4de7e: Status 404 returned error can't find the container with id e6b01f51029511d9357217ea3159791eacb6e142be46acb95b2d66431ea4de7e Nov 03 06:59:37.016645 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:37.016613 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:37.016817 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:37.016676 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:37.021046 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:37.021024 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:37.092920 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:37.092881 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c495b4dd7-thgpm" event={"ID":"84852c8e-a22b-40c8-b495-dec0174ef5d1","Type":"ContainerStarted","Data":"752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166"} Nov 03 06:59:37.093299 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:37.092939 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c495b4dd7-thgpm" event={"ID":"84852c8e-a22b-40c8-b495-dec0174ef5d1","Type":"ContainerStarted","Data":"e6b01f51029511d9357217ea3159791eacb6e142be46acb95b2d66431ea4de7e"} Nov 03 06:59:37.094064 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:37.094039 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerStarted","Data":"7964052b8e0bb8e47060806f6250e5f9681428215e9b49a157a48edb110ea28e"} Nov 03 06:59:37.098086 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:37.098066 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 06:59:37.264676 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:37.264629 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c495b4dd7-thgpm" podStartSLOduration=2.264615798 podStartE2EDuration="2.264615798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:37.263848247 +0000 UTC m=+32.456389997" watchObservedRunningTime="2026-11-03 06:59:37.264615798 +0000 UTC m=+32.457157549" Nov 03 06:59:38.772808 ip-10-0-138-121 kubenswrapper[4409]: E1103 06:59:38.772771 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 06:59:39.104578 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:39.104545 4409 generic.go:356] "Generic (PLEG): container finished" podID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerID="633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b" exitCode=0 Nov 03 06:59:39.104762 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:39.104602 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerDied","Data":"633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b"} Nov 03 06:59:39.476140 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:39.476044 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9gjvc" Nov 03 06:59:41.114751 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:41.114699 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerStarted","Data":"05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599"} Nov 03 06:59:41.115088 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:41.114809 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerStarted","Data":"5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46"} Nov 03 06:59:41.115088 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:41.114847 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerStarted","Data":"68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9"} Nov 03 06:59:41.115088 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:41.114887 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerStarted","Data":"e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45"} Nov 03 06:59:42.123285 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.123245 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerStarted","Data":"ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8"} Nov 03 06:59:42.709060 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.709012 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c495b4dd7-thgpm"] Nov 03 06:59:42.757612 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.757575 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fc4ff94d4-vx66p"] Nov 03 06:59:42.759491 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.759470 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.814643 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.814601 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fc4ff94d4-vx66p"] Nov 03 06:59:42.844872 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.844843 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-oauth-config\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.844872 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.844877 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/add150f7-6577-4cfa-9eb3-f03c219d2886-tmp\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.845123 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.844900 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-console-config\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.845123 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.844959 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-trusted-ca-bundle\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.845123 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.845020 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-serving-cert\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.845123 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.845049 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-service-ca\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.845123 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.845092 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-oauth-serving-cert\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.845320 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.845151 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpzsn\" (UniqueName: \"kubernetes.io/projected/add150f7-6577-4cfa-9eb3-f03c219d2886-kube-api-access-tpzsn\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.945786 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.945750 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-service-ca\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.945786 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.945792 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-oauth-serving-cert\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.945786 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.945808 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzsn\" (UniqueName: \"kubernetes.io/projected/add150f7-6577-4cfa-9eb3-f03c219d2886-kube-api-access-tpzsn\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.946147 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.945834 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-oauth-config\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.946147 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.945847 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/add150f7-6577-4cfa-9eb3-f03c219d2886-tmp\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.946147 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.945879 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-console-config\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.946147 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.946047 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-trusted-ca-bundle\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.946147 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.946126 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-serving-cert\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.946647 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.946566 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/add150f7-6577-4cfa-9eb3-f03c219d2886-tmp\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.947359 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.946841 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-service-ca\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.947359 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.947025 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-oauth-serving-cert\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.947736 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.947693 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-console-config\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.948129 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.948106 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-trusted-ca-bundle\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.952440 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.952414 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-serving-cert\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:42.952528 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:42.952414 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-oauth-config\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:43.011145 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:43.011070 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpzsn\" (UniqueName: \"kubernetes.io/projected/add150f7-6577-4cfa-9eb3-f03c219d2886-kube-api-access-tpzsn\") pod \"console-7fc4ff94d4-vx66p\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:43.069726 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:43.069670 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:43.134258 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:43.134227 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerStarted","Data":"933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e"} Nov 03 06:59:43.276223 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:43.276123 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.546043561 podStartE2EDuration="8.276110124s" totalImagesPullingTime="4.723494718s" totalInitContainerRuntime="6.571845ms" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:43.274862209 +0000 UTC m=+38.467403961" watchObservedRunningTime="2026-11-03 06:59:43.276110124 +0000 UTC m=+38.468651914" Nov 03 06:59:43.457132 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:43.457098 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fc4ff94d4-vx66p"] Nov 03 06:59:43.461838 ip-10-0-138-121 kubenswrapper[4409]: W1103 06:59:43.460737 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd150f7_6577_4cfa_9eb3_f03c219d2886.slice/crio-2444d65a7f1db2b963c69f2fcec60e4f4f7f7eecd28d7c9c89c28836309da8d1 WatchSource:0}: Error finding container 2444d65a7f1db2b963c69f2fcec60e4f4f7f7eecd28d7c9c89c28836309da8d1: Status 404 returned error can't find the container with id 2444d65a7f1db2b963c69f2fcec60e4f4f7f7eecd28d7c9c89c28836309da8d1 Nov 03 06:59:44.139264 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:44.139228 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fc4ff94d4-vx66p" event={"ID":"add150f7-6577-4cfa-9eb3-f03c219d2886","Type":"ContainerStarted","Data":"53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72"} Nov 03 06:59:44.139800 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:44.139278 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fc4ff94d4-vx66p" event={"ID":"add150f7-6577-4cfa-9eb3-f03c219d2886","Type":"ContainerStarted","Data":"2444d65a7f1db2b963c69f2fcec60e4f4f7f7eecd28d7c9c89c28836309da8d1"} Nov 03 06:59:44.262778 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:44.262548 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7fc4ff94d4-vx66p" podStartSLOduration=2.262531562 podStartE2EDuration="2.262531562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:59:44.262014201 +0000 UTC m=+39.454555951" watchObservedRunningTime="2026-11-03 06:59:44.262531562 +0000 UTC m=+39.455073312" Nov 03 06:59:46.385492 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:46.385459 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 06:59:53.070270 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:53.070233 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:53.070776 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:53.070298 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:53.074886 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:53.074865 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:53.178073 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:53.178044 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 06:59:53.379808 ip-10-0-138-121 kubenswrapper[4409]: I1103 06:59:53.379734 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57999f5f57-f8mz2"] Nov 03 07:00:07.729101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:07.729034 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c495b4dd7-thgpm" podUID="84852c8e-a22b-40c8-b495-dec0174ef5d1" containerName="console" containerID="cri-o://752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166" gracePeriod=15 Nov 03 07:00:07.970797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:07.970775 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 07:00:08.128109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128079 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-config\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-config\") pod \"84852c8e-a22b-40c8-b495-dec0174ef5d1\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " Nov 03 07:00:08.128109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128110 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84852c8e-a22b-40c8-b495-dec0174ef5d1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/84852c8e-a22b-40c8-b495-dec0174ef5d1-tmp\") pod \"84852c8e-a22b-40c8-b495-dec0174ef5d1\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " Nov 03 07:00:08.128331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128131 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-trusted-ca-bundle\") pod \"84852c8e-a22b-40c8-b495-dec0174ef5d1\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " Nov 03 07:00:08.128331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128161 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-oauth-config\") pod \"84852c8e-a22b-40c8-b495-dec0174ef5d1\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " Nov 03 07:00:08.128331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128185 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-serving-cert\") pod \"84852c8e-a22b-40c8-b495-dec0174ef5d1\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " Nov 03 07:00:08.128331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128213 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-service-ca\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-service-ca\") pod \"84852c8e-a22b-40c8-b495-dec0174ef5d1\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " Nov 03 07:00:08.128540 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128339 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84852c8e-a22b-40c8-b495-dec0174ef5d1-kube-api-access-jb8qw\" (UniqueName: \"kubernetes.io/projected/84852c8e-a22b-40c8-b495-dec0174ef5d1-kube-api-access-jb8qw\") pod \"84852c8e-a22b-40c8-b495-dec0174ef5d1\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " Nov 03 07:00:08.128540 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128402 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-oauth-serving-cert\") pod \"84852c8e-a22b-40c8-b495-dec0174ef5d1\" (UID: \"84852c8e-a22b-40c8-b495-dec0174ef5d1\") " Nov 03 07:00:08.128540 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128399 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84852c8e-a22b-40c8-b495-dec0174ef5d1-tmp" pod "84852c8e-a22b-40c8-b495-dec0174ef5d1" (UID: "84852c8e-a22b-40c8-b495-dec0174ef5d1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:00:08.128540 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128491 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-config" pod "84852c8e-a22b-40c8-b495-dec0174ef5d1" (UID: "84852c8e-a22b-40c8-b495-dec0174ef5d1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:00:08.128762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128597 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-trusted-ca-bundle" pod "84852c8e-a22b-40c8-b495-dec0174ef5d1" (UID: "84852c8e-a22b-40c8-b495-dec0174ef5d1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:00:08.128819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128766 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-service-ca" pod "84852c8e-a22b-40c8-b495-dec0174ef5d1" (UID: "84852c8e-a22b-40c8-b495-dec0174ef5d1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:00:08.128873 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128834 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-oauth-serving-cert" pod "84852c8e-a22b-40c8-b495-dec0174ef5d1" (UID: "84852c8e-a22b-40c8-b495-dec0174ef5d1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:00:08.128873 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128848 4409 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-config\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:08.128873 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128868 4409 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84852c8e-a22b-40c8-b495-dec0174ef5d1-tmp\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:08.129047 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128884 4409 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-trusted-ca-bundle\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:08.129047 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.128898 4409 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:08.130596 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.130578 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-oauth-config" pod "84852c8e-a22b-40c8-b495-dec0174ef5d1" (UID: "84852c8e-a22b-40c8-b495-dec0174ef5d1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:08.130670 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.130637 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84852c8e-a22b-40c8-b495-dec0174ef5d1-kube-api-access-jb8qw" pod "84852c8e-a22b-40c8-b495-dec0174ef5d1" (UID: "84852c8e-a22b-40c8-b495-dec0174ef5d1"). InnerVolumeSpecName "kube-api-access-jb8qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:00:08.130706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.130669 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-serving-cert" pod "84852c8e-a22b-40c8-b495-dec0174ef5d1" (UID: "84852c8e-a22b-40c8-b495-dec0174ef5d1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:08.225817 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.225794 4409 generic.go:356] "Generic (PLEG): container finished" podID="84852c8e-a22b-40c8-b495-dec0174ef5d1" containerID="752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166" exitCode=0 Nov 03 07:00:08.225927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.225850 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c495b4dd7-thgpm" Nov 03 07:00:08.225927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.225871 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c495b4dd7-thgpm" event={"ID":"84852c8e-a22b-40c8-b495-dec0174ef5d1","Type":"ContainerDied","Data":"752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166"} Nov 03 07:00:08.225927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.225908 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c495b4dd7-thgpm" event={"ID":"84852c8e-a22b-40c8-b495-dec0174ef5d1","Type":"ContainerDied","Data":"e6b01f51029511d9357217ea3159791eacb6e142be46acb95b2d66431ea4de7e"} Nov 03 07:00:08.226048 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.225934 4409 scope.go:122] "RemoveContainer" containerID="752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166" Nov 03 07:00:08.229483 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.229464 4409 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-oauth-config\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:08.229553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.229491 4409 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84852c8e-a22b-40c8-b495-dec0174ef5d1-console-serving-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:08.229553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.229508 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb8qw\" (UniqueName: \"kubernetes.io/projected/84852c8e-a22b-40c8-b495-dec0174ef5d1-kube-api-access-jb8qw\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:08.229553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.229525 4409 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84852c8e-a22b-40c8-b495-dec0174ef5d1-oauth-serving-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:08.234130 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.234113 4409 scope.go:122] "RemoveContainer" containerID="752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166" Nov 03 07:00:08.234418 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:00:08.234391 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166\": container with ID starting with 752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166 not found: ID does not exist" containerID="752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166" Nov 03 07:00:08.234497 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.234432 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166"} err="failed to get container status \"752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166\": rpc error: code = NotFound desc = could not find container \"752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166\": container with ID starting with 752893a619c643b38d1e9c42b6dae3ee3523f89a448ce06a44033f0263caa166 not found: ID does not exist" Nov 03 07:00:08.393906 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.393847 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c495b4dd7-thgpm"] Nov 03 07:00:08.399972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.399953 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c495b4dd7-thgpm"] Nov 03 07:00:08.771885 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:00:08.771812 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:00:08.909841 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:08.909803 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84852c8e-a22b-40c8-b495-dec0174ef5d1" path="/var/lib/kubelet/pods/84852c8e-a22b-40c8-b495-dec0174ef5d1/volumes" Nov 03 07:00:18.399290 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.399221 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-57999f5f57-f8mz2" podUID="40548b01-8d84-4461-b36a-92fdafcb75c5" containerName="console" containerID="cri-o://474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657" gracePeriod=15 Nov 03 07:00:18.652883 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.652836 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 07:00:18.800904 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.800880 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-console-config\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-console-config\") pod \"40548b01-8d84-4461-b36a-92fdafcb75c5\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " Nov 03 07:00:18.801027 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.800918 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40548b01-8d84-4461-b36a-92fdafcb75c5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/40548b01-8d84-4461-b36a-92fdafcb75c5-tmp\") pod \"40548b01-8d84-4461-b36a-92fdafcb75c5\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " Nov 03 07:00:18.801027 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.800946 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-oauth-serving-cert\") pod \"40548b01-8d84-4461-b36a-92fdafcb75c5\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " Nov 03 07:00:18.801027 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.800979 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-oauth-config\") pod \"40548b01-8d84-4461-b36a-92fdafcb75c5\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " Nov 03 07:00:18.801027 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.801009 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-serving-cert\") pod \"40548b01-8d84-4461-b36a-92fdafcb75c5\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " Nov 03 07:00:18.801210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.801036 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40548b01-8d84-4461-b36a-92fdafcb75c5-kube-api-access-9fnm6\" (UniqueName: \"kubernetes.io/projected/40548b01-8d84-4461-b36a-92fdafcb75c5-kube-api-access-9fnm6\") pod \"40548b01-8d84-4461-b36a-92fdafcb75c5\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " Nov 03 07:00:18.801210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.801066 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-service-ca\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-service-ca\") pod \"40548b01-8d84-4461-b36a-92fdafcb75c5\" (UID: \"40548b01-8d84-4461-b36a-92fdafcb75c5\") " Nov 03 07:00:18.801210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.801177 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40548b01-8d84-4461-b36a-92fdafcb75c5-tmp" pod "40548b01-8d84-4461-b36a-92fdafcb75c5" (UID: "40548b01-8d84-4461-b36a-92fdafcb75c5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:00:18.801362 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.801328 4409 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40548b01-8d84-4461-b36a-92fdafcb75c5-tmp\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:18.801433 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.801301 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-console-config" pod "40548b01-8d84-4461-b36a-92fdafcb75c5" (UID: "40548b01-8d84-4461-b36a-92fdafcb75c5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:00:18.801491 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.801476 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-oauth-serving-cert" pod "40548b01-8d84-4461-b36a-92fdafcb75c5" (UID: "40548b01-8d84-4461-b36a-92fdafcb75c5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:00:18.801549 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.801526 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-service-ca" pod "40548b01-8d84-4461-b36a-92fdafcb75c5" (UID: "40548b01-8d84-4461-b36a-92fdafcb75c5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:00:18.803291 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.803265 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-serving-cert" pod "40548b01-8d84-4461-b36a-92fdafcb75c5" (UID: "40548b01-8d84-4461-b36a-92fdafcb75c5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:18.803393 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.803281 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-oauth-config" pod "40548b01-8d84-4461-b36a-92fdafcb75c5" (UID: "40548b01-8d84-4461-b36a-92fdafcb75c5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:18.803393 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.803377 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40548b01-8d84-4461-b36a-92fdafcb75c5-kube-api-access-9fnm6" pod "40548b01-8d84-4461-b36a-92fdafcb75c5" (UID: "40548b01-8d84-4461-b36a-92fdafcb75c5"). InnerVolumeSpecName "kube-api-access-9fnm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:00:18.902591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.902565 4409 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-oauth-serving-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:18.902591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.902588 4409 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-oauth-config\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:18.902751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.902602 4409 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/40548b01-8d84-4461-b36a-92fdafcb75c5-console-serving-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:18.902751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.902616 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fnm6\" (UniqueName: \"kubernetes.io/projected/40548b01-8d84-4461-b36a-92fdafcb75c5-kube-api-access-9fnm6\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:18.902751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.902629 4409 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:18.902751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:18.902641 4409 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/40548b01-8d84-4461-b36a-92fdafcb75c5-console-config\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:19.264677 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:19.264599 4409 generic.go:356] "Generic (PLEG): container finished" podID="40548b01-8d84-4461-b36a-92fdafcb75c5" containerID="474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657" exitCode=0 Nov 03 07:00:19.264677 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:19.264650 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57999f5f57-f8mz2" Nov 03 07:00:19.264677 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:19.264662 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57999f5f57-f8mz2" event={"ID":"40548b01-8d84-4461-b36a-92fdafcb75c5","Type":"ContainerDied","Data":"474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657"} Nov 03 07:00:19.264915 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:19.264702 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57999f5f57-f8mz2" event={"ID":"40548b01-8d84-4461-b36a-92fdafcb75c5","Type":"ContainerDied","Data":"a4e01616e59df23b01da0c704a1666b3c15ab1cd0ad4e5b3bf6abc605bde287b"} Nov 03 07:00:19.264915 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:19.264742 4409 scope.go:122] "RemoveContainer" containerID="474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657" Nov 03 07:00:19.272648 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:19.272630 4409 scope.go:122] "RemoveContainer" containerID="474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657" Nov 03 07:00:19.272955 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:00:19.272936 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657\": container with ID starting with 474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657 not found: ID does not exist" containerID="474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657" Nov 03 07:00:19.273007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:19.272965 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657"} err="failed to get container status \"474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657\": rpc error: code = NotFound desc = could not find container \"474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657\": container with ID starting with 474891eea847b21ab4d87e098df3f65f6204352121580c4f342cc6eeb05ff657 not found: ID does not exist" Nov 03 07:00:19.445299 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:19.445271 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57999f5f57-f8mz2"] Nov 03 07:00:19.494782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:19.494757 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57999f5f57-f8mz2"] Nov 03 07:00:20.910177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:20.910145 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40548b01-8d84-4461-b36a-92fdafcb75c5" path="/var/lib/kubelet/pods/40548b01-8d84-4461-b36a-92fdafcb75c5/volumes" Nov 03 07:00:38.772889 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:00:38.772855 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:00:39.386571 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:39.386538 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 07:00:39.386963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:39.386933 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="alertmanager" containerID="cri-o://e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45" gracePeriod=120 Nov 03 07:00:39.387101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:39.386983 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="kube-rbac-proxy-metric" containerID="cri-o://ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8" gracePeriod=120 Nov 03 07:00:39.387101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:39.387021 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="prom-label-proxy" containerID="cri-o://933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e" gracePeriod=120 Nov 03 07:00:39.387101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:39.387048 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="kube-rbac-proxy-web" containerID="cri-o://5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46" gracePeriod=120 Nov 03 07:00:39.387251 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:39.387071 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="kube-rbac-proxy" containerID="cri-o://05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599" gracePeriod=120 Nov 03 07:00:39.387251 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:39.387098 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="config-reloader" containerID="cri-o://68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9" gracePeriod=120 Nov 03 07:00:40.338174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.338141 4409 generic.go:356] "Generic (PLEG): container finished" podID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerID="933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e" exitCode=0 Nov 03 07:00:40.338174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.338166 4409 generic.go:356] "Generic (PLEG): container finished" podID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerID="05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599" exitCode=0 Nov 03 07:00:40.338174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.338172 4409 generic.go:356] "Generic (PLEG): container finished" podID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerID="68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9" exitCode=0 Nov 03 07:00:40.338174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.338178 4409 generic.go:356] "Generic (PLEG): container finished" podID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerID="e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45" exitCode=0 Nov 03 07:00:40.338655 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.338214 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerDied","Data":"933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e"} Nov 03 07:00:40.338655 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.338287 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerDied","Data":"05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599"} Nov 03 07:00:40.338655 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.338314 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerDied","Data":"68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9"} Nov 03 07:00:40.338655 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.338338 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerDied","Data":"e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45"} Nov 03 07:00:40.649999 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.649974 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:40.754343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754305 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-kube-api-access-87lc7\" (UniqueName: \"kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-kube-api-access-87lc7\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.754539 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754351 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-cluster-tls-config\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.754539 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754386 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.754539 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754423 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-tls-assets\" (UniqueName: \"kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-tls-assets\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.754539 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754451 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-out\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.754539 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754473 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-web-config\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-web-config\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.754539 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754506 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-volume\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-volume\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.754891 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754551 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-main-db\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.754891 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754589 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-trusted-ca-bundle\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.754891 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754629 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.754891 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754659 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-main-tls\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.754891 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754745 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.754891 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.754776 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-metrics-client-ca\") pod \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\" (UID: \"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e\") " Nov 03 07:00:40.755306 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.755199 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-trusted-ca-bundle" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:00:40.755306 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.755276 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-metrics-client-ca" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:00:40.756485 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.756455 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-main-db" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:00:40.758131 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.758088 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-tls-assets" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:00:40.758239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.758220 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-metric" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:40.758432 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.758373 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-volume" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:40.759007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.758972 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-main-tls" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:40.759282 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.759146 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-kube-api-access-87lc7" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "kube-api-access-87lc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:00:40.759282 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.759245 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-out" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:00:40.759655 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.759635 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-web" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:40.760093 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.760068 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:40.762472 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.762447 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-cluster-tls-config" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:40.768514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.768490 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-web-config" pod "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" (UID: "9c4f27a2-ff03-46a0-b793-9210ac6f4e7e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:00:40.855419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855384 4409 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:40.855419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855412 4409 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:40.855419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855421 4409 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-main-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:40.855419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855430 4409 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:40.855691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855441 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-metrics-client-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:40.855691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855451 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87lc7\" (UniqueName: \"kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-kube-api-access-87lc7\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:40.855691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855460 4409 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-cluster-tls-config\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:40.855691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855470 4409 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:40.855691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855479 4409 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-tls-assets\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:40.855691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855487 4409 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-out\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:40.855691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855498 4409 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-web-config\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:40.855691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855507 4409 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:40.855691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:40.855515 4409 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e-alertmanager-main-db\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:00:41.345014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.344982 4409 generic.go:356] "Generic (PLEG): container finished" podID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerID="ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8" exitCode=0 Nov 03 07:00:41.345014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.345007 4409 generic.go:356] "Generic (PLEG): container finished" podID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerID="5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46" exitCode=0 Nov 03 07:00:41.345444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.345066 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.345444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.345067 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerDied","Data":"ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8"} Nov 03 07:00:41.345444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.345153 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerDied","Data":"5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46"} Nov 03 07:00:41.345444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.345197 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c4f27a2-ff03-46a0-b793-9210ac6f4e7e","Type":"ContainerDied","Data":"7964052b8e0bb8e47060806f6250e5f9681428215e9b49a157a48edb110ea28e"} Nov 03 07:00:41.345444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.345222 4409 scope.go:122] "RemoveContainer" containerID="933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e" Nov 03 07:00:41.352694 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.352675 4409 scope.go:122] "RemoveContainer" containerID="ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8" Nov 03 07:00:41.359656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.359637 4409 scope.go:122] "RemoveContainer" containerID="05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599" Nov 03 07:00:41.366605 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.366585 4409 scope.go:122] "RemoveContainer" containerID="5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46" Nov 03 07:00:41.373121 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.373103 4409 scope.go:122] "RemoveContainer" containerID="68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9" Nov 03 07:00:41.379895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.379877 4409 scope.go:122] "RemoveContainer" containerID="e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45" Nov 03 07:00:41.386320 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.386300 4409 scope.go:122] "RemoveContainer" containerID="633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b" Nov 03 07:00:41.392852 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.392836 4409 scope.go:122] "RemoveContainer" containerID="933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e" Nov 03 07:00:41.393113 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:00:41.393087 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e\": container with ID starting with 933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e not found: ID does not exist" containerID="933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e" Nov 03 07:00:41.393193 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.393122 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e"} err="failed to get container status \"933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e\": rpc error: code = NotFound desc = could not find container \"933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e\": container with ID starting with 933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e not found: ID does not exist" Nov 03 07:00:41.393193 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.393149 4409 scope.go:122] "RemoveContainer" containerID="ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8" Nov 03 07:00:41.393388 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:00:41.393370 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8\": container with ID starting with ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8 not found: ID does not exist" containerID="ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8" Nov 03 07:00:41.393434 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.393395 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8"} err="failed to get container status \"ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8\": rpc error: code = NotFound desc = could not find container \"ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8\": container with ID starting with ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8 not found: ID does not exist" Nov 03 07:00:41.393434 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.393410 4409 scope.go:122] "RemoveContainer" containerID="05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599" Nov 03 07:00:41.393647 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:00:41.393629 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599\": container with ID starting with 05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599 not found: ID does not exist" containerID="05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599" Nov 03 07:00:41.393685 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.393653 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599"} err="failed to get container status \"05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599\": rpc error: code = NotFound desc = could not find container \"05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599\": container with ID starting with 05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599 not found: ID does not exist" Nov 03 07:00:41.393685 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.393672 4409 scope.go:122] "RemoveContainer" containerID="5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46" Nov 03 07:00:41.393905 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:00:41.393889 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46\": container with ID starting with 5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46 not found: ID does not exist" containerID="5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46" Nov 03 07:00:41.393960 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.393908 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46"} err="failed to get container status \"5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46\": rpc error: code = NotFound desc = could not find container \"5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46\": container with ID starting with 5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46 not found: ID does not exist" Nov 03 07:00:41.393960 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.393919 4409 scope.go:122] "RemoveContainer" containerID="68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9" Nov 03 07:00:41.394139 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:00:41.394119 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9\": container with ID starting with 68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9 not found: ID does not exist" containerID="68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9" Nov 03 07:00:41.394202 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.394147 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9"} err="failed to get container status \"68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9\": rpc error: code = NotFound desc = could not find container \"68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9\": container with ID starting with 68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9 not found: ID does not exist" Nov 03 07:00:41.394202 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.394168 4409 scope.go:122] "RemoveContainer" containerID="e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45" Nov 03 07:00:41.394407 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:00:41.394393 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45\": container with ID starting with e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45 not found: ID does not exist" containerID="e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45" Nov 03 07:00:41.394445 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.394411 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45"} err="failed to get container status \"e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45\": rpc error: code = NotFound desc = could not find container \"e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45\": container with ID starting with e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45 not found: ID does not exist" Nov 03 07:00:41.394445 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.394422 4409 scope.go:122] "RemoveContainer" containerID="633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b" Nov 03 07:00:41.394627 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:00:41.394610 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b\": container with ID starting with 633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b not found: ID does not exist" containerID="633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b" Nov 03 07:00:41.394690 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.394634 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b"} err="failed to get container status \"633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b\": rpc error: code = NotFound desc = could not find container \"633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b\": container with ID starting with 633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b not found: ID does not exist" Nov 03 07:00:41.394690 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.394652 4409 scope.go:122] "RemoveContainer" containerID="933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e" Nov 03 07:00:41.394925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.394904 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e"} err="failed to get container status \"933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e\": rpc error: code = NotFound desc = could not find container \"933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e\": container with ID starting with 933b50913f558126aac1a7c041ca9be2ccc0301b66d3c46b9f1e3e0656f96f0e not found: ID does not exist" Nov 03 07:00:41.394964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.394928 4409 scope.go:122] "RemoveContainer" containerID="ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8" Nov 03 07:00:41.395143 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.395128 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8"} err="failed to get container status \"ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8\": rpc error: code = NotFound desc = could not find container \"ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8\": container with ID starting with ec2f742b3047f745b5c73c6528b34a4282c736ec7d87d28b31ce46166d35a5e8 not found: ID does not exist" Nov 03 07:00:41.395184 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.395143 4409 scope.go:122] "RemoveContainer" containerID="05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599" Nov 03 07:00:41.395387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.395370 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599"} err="failed to get container status \"05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599\": rpc error: code = NotFound desc = could not find container \"05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599\": container with ID starting with 05d044cae39c5ed6dcbcb2f7aace38dcbd53d28f0b59c9f421b38da010cab599 not found: ID does not exist" Nov 03 07:00:41.395387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.395387 4409 scope.go:122] "RemoveContainer" containerID="5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46" Nov 03 07:00:41.395592 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.395576 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46"} err="failed to get container status \"5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46\": rpc error: code = NotFound desc = could not find container \"5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46\": container with ID starting with 5df3f943f5edb41be8bb6cd128da2b5aee2d0240cb83e217dfb3778ebfde9f46 not found: ID does not exist" Nov 03 07:00:41.395632 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.395591 4409 scope.go:122] "RemoveContainer" containerID="68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9" Nov 03 07:00:41.395847 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.395828 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9"} err="failed to get container status \"68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9\": rpc error: code = NotFound desc = could not find container \"68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9\": container with ID starting with 68118aaa83d40aa594eb2414a210ef4c6472e80a62f2acb14b37130c74d3dbb9 not found: ID does not exist" Nov 03 07:00:41.395917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.395847 4409 scope.go:122] "RemoveContainer" containerID="e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45" Nov 03 07:00:41.396087 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.396071 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45"} err="failed to get container status \"e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45\": rpc error: code = NotFound desc = could not find container \"e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45\": container with ID starting with e519220b6f9fe1a90f83e3be7d87842218c44eabcbd37f5aa37058a7f727fb45 not found: ID does not exist" Nov 03 07:00:41.396147 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.396088 4409 scope.go:122] "RemoveContainer" containerID="633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b" Nov 03 07:00:41.396302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.396286 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b"} err="failed to get container status \"633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b\": rpc error: code = NotFound desc = could not find container \"633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b\": container with ID starting with 633333d0a4e8564ab87b25ba53641fe71d22ce4fa189bfa8a8f8cfe44318786b not found: ID does not exist" Nov 03 07:00:41.516985 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.516952 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 07:00:41.566938 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.566895 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 07:00:41.579748 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.579702 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 07:00:41.579957 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.579946 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="kube-rbac-proxy-metric" Nov 03 07:00:41.579997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.579959 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="kube-rbac-proxy-metric" Nov 03 07:00:41.579997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.579972 4409 cpu_manager.go:414] "containerMap: removing container" podUID="84852c8e-a22b-40c8-b495-dec0174ef5d1" containerName="console" Nov 03 07:00:41.579997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.579976 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84852c8e-a22b-40c8-b495-dec0174ef5d1" containerName="console" Nov 03 07:00:41.579997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.579988 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="prom-label-proxy" Nov 03 07:00:41.579997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.579994 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="prom-label-proxy" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580002 4409 cpu_manager.go:414] "containerMap: removing container" podUID="40548b01-8d84-4461-b36a-92fdafcb75c5" containerName="console" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580007 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40548b01-8d84-4461-b36a-92fdafcb75c5" containerName="console" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580013 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="alertmanager" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580018 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="alertmanager" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580027 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="config-reloader" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580031 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="config-reloader" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580036 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="kube-rbac-proxy-web" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580041 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="kube-rbac-proxy-web" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580050 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="init-config-reloader" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580054 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="init-config-reloader" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580061 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="kube-rbac-proxy" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580065 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="kube-rbac-proxy" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580100 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="config-reloader" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580106 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="kube-rbac-proxy-web" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580112 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="84852c8e-a22b-40c8-b495-dec0174ef5d1" containerName="console" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580118 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="kube-rbac-proxy" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580122 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="prom-label-proxy" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580127 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="40548b01-8d84-4461-b36a-92fdafcb75c5" containerName="console" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580131 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="alertmanager" Nov 03 07:00:41.580138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.580136 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" containerName="kube-rbac-proxy-metric" Nov 03 07:00:41.584859 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.584844 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.642379 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.642258 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 07:00:41.642379 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.642278 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 07:00:41.642379 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.642289 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 07:00:41.642379 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.642323 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 07:00:41.642379 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.642290 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 07:00:41.642697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.642415 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 07:00:41.642697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.642537 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 07:00:41.642697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.642538 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 07:00:41.642697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.642561 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tjb9h\"" Nov 03 07:00:41.663112 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663073 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06bbffc1-4081-4059-8b5c-40fb6567a736-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.663304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663119 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06bbffc1-4081-4059-8b5c-40fb6567a736-config-out\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.663304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663147 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-web-config\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.663304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663248 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.663304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663287 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.663509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663313 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-config-volume\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.663509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663368 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk2jd\" (UniqueName: \"kubernetes.io/projected/06bbffc1-4081-4059-8b5c-40fb6567a736-kube-api-access-sk2jd\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.663509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663405 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06bbffc1-4081-4059-8b5c-40fb6567a736-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.663509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663459 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06bbffc1-4081-4059-8b5c-40fb6567a736-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.663509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663493 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06bbffc1-4081-4059-8b5c-40fb6567a736-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.663695 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663541 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.663695 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663559 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.663695 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.663582 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.691107 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.691079 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 07:00:41.693311 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.693287 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 07:00:41.764642 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.764611 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06bbffc1-4081-4059-8b5c-40fb6567a736-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.764642 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.764644 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06bbffc1-4081-4059-8b5c-40fb6567a736-config-out\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.764936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.764665 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-web-config\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.764936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.764696 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.764936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.764788 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.764936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.764830 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-config-volume\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.764936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.764856 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk2jd\" (UniqueName: \"kubernetes.io/projected/06bbffc1-4081-4059-8b5c-40fb6567a736-kube-api-access-sk2jd\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.764936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.764887 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06bbffc1-4081-4059-8b5c-40fb6567a736-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.764936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.764910 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06bbffc1-4081-4059-8b5c-40fb6567a736-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.765283 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.764966 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06bbffc1-4081-4059-8b5c-40fb6567a736-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.765283 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.764992 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.765283 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.765016 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.765283 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.765046 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.766292 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.765937 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06bbffc1-4081-4059-8b5c-40fb6567a736-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.766292 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.766029 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06bbffc1-4081-4059-8b5c-40fb6567a736-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.766506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.766416 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06bbffc1-4081-4059-8b5c-40fb6567a736-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.768199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.767812 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06bbffc1-4081-4059-8b5c-40fb6567a736-config-out\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.768199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.768072 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.768199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.768143 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-web-config\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.768199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.768191 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.768881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.768861 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.768881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.768866 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-config-volume\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.769186 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.769166 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06bbffc1-4081-4059-8b5c-40fb6567a736-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.769586 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.769567 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.769999 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.769983 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06bbffc1-4081-4059-8b5c-40fb6567a736-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.829906 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.829874 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk2jd\" (UniqueName: \"kubernetes.io/projected/06bbffc1-4081-4059-8b5c-40fb6567a736-kube-api-access-sk2jd\") pod \"alertmanager-main-0\" (UID: \"06bbffc1-4081-4059-8b5c-40fb6567a736\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:41.910998 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:41.910917 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 07:00:42.393645 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:42.393614 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 07:00:42.423307 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:00:42.423275 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bbffc1_4081_4059_8b5c_40fb6567a736.slice/crio-59937e9f831dfa2621f6dc53dabe4b6055b944ab80e540b6fb4741e019f832a8 WatchSource:0}: Error finding container 59937e9f831dfa2621f6dc53dabe4b6055b944ab80e540b6fb4741e019f832a8: Status 404 returned error can't find the container with id 59937e9f831dfa2621f6dc53dabe4b6055b944ab80e540b6fb4741e019f832a8 Nov 03 07:00:42.910910 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:42.910876 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4f27a2-ff03-46a0-b793-9210ac6f4e7e" path="/var/lib/kubelet/pods/9c4f27a2-ff03-46a0-b793-9210ac6f4e7e/volumes" Nov 03 07:00:43.037314 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.037275 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8649454f66-sldzs"] Nov 03 07:00:43.040793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.040770 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.077669 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.077640 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddk5p\" (UniqueName: \"kubernetes.io/projected/1a26273e-e854-4a0b-8ef4-95a792060cf3-kube-api-access-ddk5p\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.077805 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.077683 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1a26273e-e854-4a0b-8ef4-95a792060cf3-federate-client-tls\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.077805 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.077781 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a26273e-e854-4a0b-8ef4-95a792060cf3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.077885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.077839 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a26273e-e854-4a0b-8ef4-95a792060cf3-serving-certs-ca-bundle\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.077885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.077858 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1a26273e-e854-4a0b-8ef4-95a792060cf3-telemeter-client-tls\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.077885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.077881 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a26273e-e854-4a0b-8ef4-95a792060cf3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.078025 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.077902 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1a26273e-e854-4a0b-8ef4-95a792060cf3-secret-telemeter-client\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.078080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.078029 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a26273e-e854-4a0b-8ef4-95a792060cf3-metrics-client-ca\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.095498 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.095468 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 07:00:43.179043 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.178963 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddk5p\" (UniqueName: \"kubernetes.io/projected/1a26273e-e854-4a0b-8ef4-95a792060cf3-kube-api-access-ddk5p\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.179043 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.179013 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1a26273e-e854-4a0b-8ef4-95a792060cf3-federate-client-tls\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.179043 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.179039 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a26273e-e854-4a0b-8ef4-95a792060cf3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.179327 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.179070 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a26273e-e854-4a0b-8ef4-95a792060cf3-serving-certs-ca-bundle\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.179327 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.179094 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1a26273e-e854-4a0b-8ef4-95a792060cf3-telemeter-client-tls\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.179327 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.179223 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a26273e-e854-4a0b-8ef4-95a792060cf3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.179327 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.179271 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1a26273e-e854-4a0b-8ef4-95a792060cf3-secret-telemeter-client\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.179327 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.179293 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a26273e-e854-4a0b-8ef4-95a792060cf3-metrics-client-ca\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.179695 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.179676 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a26273e-e854-4a0b-8ef4-95a792060cf3-serving-certs-ca-bundle\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.179832 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.179816 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a26273e-e854-4a0b-8ef4-95a792060cf3-metrics-client-ca\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.194681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.194564 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 07:00:43.194681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.194578 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-z559s\"" Nov 03 07:00:43.194681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.194566 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 07:00:43.194681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.194630 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 07:00:43.194681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.194644 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 07:00:43.194681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.194631 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 07:00:43.201025 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.200971 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a26273e-e854-4a0b-8ef4-95a792060cf3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.202796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.202769 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a26273e-e854-4a0b-8ef4-95a792060cf3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.202888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.202800 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1a26273e-e854-4a0b-8ef4-95a792060cf3-secret-telemeter-client\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.203162 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.203140 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1a26273e-e854-4a0b-8ef4-95a792060cf3-federate-client-tls\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.203199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.203165 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1a26273e-e854-4a0b-8ef4-95a792060cf3-telemeter-client-tls\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.242634 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.242597 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddk5p\" (UniqueName: \"kubernetes.io/projected/1a26273e-e854-4a0b-8ef4-95a792060cf3-kube-api-access-ddk5p\") pod \"telemeter-client-8649454f66-sldzs\" (UID: \"1a26273e-e854-4a0b-8ef4-95a792060cf3\") " pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.258082 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.258049 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8649454f66-sldzs"] Nov 03 07:00:43.354959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.354924 4409 generic.go:356] "Generic (PLEG): container finished" podID="06bbffc1-4081-4059-8b5c-40fb6567a736" containerID="ce0f4ef8d4abd51296692ca8014e49886bd49ac078eeb150fb3eb2077d3170ea" exitCode=0 Nov 03 07:00:43.355123 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.354976 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06bbffc1-4081-4059-8b5c-40fb6567a736","Type":"ContainerDied","Data":"ce0f4ef8d4abd51296692ca8014e49886bd49ac078eeb150fb3eb2077d3170ea"} Nov 03 07:00:43.355123 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.355024 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06bbffc1-4081-4059-8b5c-40fb6567a736","Type":"ContainerStarted","Data":"59937e9f831dfa2621f6dc53dabe4b6055b944ab80e540b6fb4741e019f832a8"} Nov 03 07:00:43.356098 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.356074 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" Nov 03 07:00:43.789572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:43.789550 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8649454f66-sldzs"] Nov 03 07:00:43.792093 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:00:43.792055 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a26273e_e854_4a0b_8ef4_95a792060cf3.slice/crio-76ed3ac4a3f4fe48016ca7e729483c058460492acf38759ee2ac3adb3162d841 WatchSource:0}: Error finding container 76ed3ac4a3f4fe48016ca7e729483c058460492acf38759ee2ac3adb3162d841: Status 404 returned error can't find the container with id 76ed3ac4a3f4fe48016ca7e729483c058460492acf38759ee2ac3adb3162d841 Nov 03 07:00:44.360249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:44.360213 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" event={"ID":"1a26273e-e854-4a0b-8ef4-95a792060cf3","Type":"ContainerStarted","Data":"76ed3ac4a3f4fe48016ca7e729483c058460492acf38759ee2ac3adb3162d841"} Nov 03 07:00:44.363296 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:44.363269 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06bbffc1-4081-4059-8b5c-40fb6567a736","Type":"ContainerStarted","Data":"91b9f75d083f24f899b92ee2c22f0dc3edccf45aa34643af80b1d72a400ae146"} Nov 03 07:00:44.363424 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:44.363336 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06bbffc1-4081-4059-8b5c-40fb6567a736","Type":"ContainerStarted","Data":"6292f8ae6767b3446088425c9e2055fab383a85c9e1c152d2cf1bf1afd7b7274"} Nov 03 07:00:44.363424 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:44.363374 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06bbffc1-4081-4059-8b5c-40fb6567a736","Type":"ContainerStarted","Data":"54e6703a71843197acaa669518a5194d7377e4917ea009fb7ed7937631bf2e7d"} Nov 03 07:00:44.363424 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:44.363412 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06bbffc1-4081-4059-8b5c-40fb6567a736","Type":"ContainerStarted","Data":"0f0a1e1367aaab149561bcc87a4e594c50b760fdaca4d44e99a750d65b506320"} Nov 03 07:00:44.363524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:44.363449 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06bbffc1-4081-4059-8b5c-40fb6567a736","Type":"ContainerStarted","Data":"035d77455c6a8a3924b4fab788e2108181f9bf3727d6c399f73f8a1708bc124e"} Nov 03 07:00:44.363524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:44.363482 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06bbffc1-4081-4059-8b5c-40fb6567a736","Type":"ContainerStarted","Data":"75b1da01992538208a6eeabf6454f04ad528ccda4860b93044ec6046913ed6a4"} Nov 03 07:00:44.509319 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:44.509264 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.496414385 podStartE2EDuration="3.509245073s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.830688ms" isStatefulPod=false podCreationTimestamp="2026-11-03 07:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:00:44.507612118 +0000 UTC m=+99.700153869" watchObservedRunningTime="2026-11-03 07:00:44.509245073 +0000 UTC m=+99.701786819" Nov 03 07:00:46.373117 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:46.373077 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" event={"ID":"1a26273e-e854-4a0b-8ef4-95a792060cf3","Type":"ContainerStarted","Data":"8dcb635aa47a329a0fdc2b224e06f3ae539504eb67949865dd2f47338f227055"} Nov 03 07:00:46.373514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:46.373159 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" event={"ID":"1a26273e-e854-4a0b-8ef4-95a792060cf3","Type":"ContainerStarted","Data":"d527022c32c4ef6562f690f3498bba69604184b1c58e02982e2cb43eb04fb1cd"} Nov 03 07:00:46.373514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:46.373179 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" event={"ID":"1a26273e-e854-4a0b-8ef4-95a792060cf3","Type":"ContainerStarted","Data":"9945145a9a96ea16dc0b2c09409a490ed4dcb5f98c26edc8981d00a179142ae0"} Nov 03 07:00:46.595857 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:46.595809 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8649454f66-sldzs" podStartSLOduration=2.494463477 podStartE2EDuration="4.595796189s" totalImagesPullingTime="2.101332712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:00:46.546182775 +0000 UTC m=+101.738724526" watchObservedRunningTime="2026-11-03 07:00:46.595796189 +0000 UTC m=+101.788337939" Nov 03 07:00:47.002437 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.002404 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fbd4d94c4-qgm6l"] Nov 03 07:00:47.005467 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.005450 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.067391 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.067362 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbd4d94c4-qgm6l"] Nov 03 07:00:47.111904 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.111874 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-serving-cert\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.112008 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.111908 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grxqk\" (UniqueName: \"kubernetes.io/projected/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-kube-api-access-grxqk\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.112008 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.111934 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-service-ca\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.112083 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.112034 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-tmp\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.112083 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.112065 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-config\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.112146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.112084 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-oauth-config\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.112146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.112109 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-oauth-serving-cert\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.112228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.112210 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-trusted-ca-bundle\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.213130 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.213101 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-tmp\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.213248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.213134 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-config\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.213248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.213153 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-oauth-config\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.213248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.213168 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-oauth-serving-cert\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.213248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.213210 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-trusted-ca-bundle\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.213248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.213235 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-serving-cert\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.213520 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.213263 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grxqk\" (UniqueName: \"kubernetes.io/projected/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-kube-api-access-grxqk\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.213520 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.213335 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-service-ca\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.213625 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.213537 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-tmp\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.213985 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.213965 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-config\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.214106 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.213965 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-oauth-serving-cert\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.214106 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.214068 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-service-ca\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.214179 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.214152 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-trusted-ca-bundle\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.216397 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.216380 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-oauth-config\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.216599 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.216574 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-serving-cert\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.279914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.279851 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grxqk\" (UniqueName: \"kubernetes.io/projected/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-kube-api-access-grxqk\") pod \"console-6fbd4d94c4-qgm6l\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.316571 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.316533 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:47.698683 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:47.698650 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbd4d94c4-qgm6l"] Nov 03 07:00:47.730523 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:00:47.730499 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ff8095_af7a_446d_a4d6_a3da3d7cc02f.slice/crio-1f78d6f4e49de71a4c52946e14368c63e323b750d3e48464607d3cc25ca554e9 WatchSource:0}: Error finding container 1f78d6f4e49de71a4c52946e14368c63e323b750d3e48464607d3cc25ca554e9: Status 404 returned error can't find the container with id 1f78d6f4e49de71a4c52946e14368c63e323b750d3e48464607d3cc25ca554e9 Nov 03 07:00:48.384765 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:48.384728 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbd4d94c4-qgm6l" event={"ID":"82ff8095-af7a-446d-a4d6-a3da3d7cc02f","Type":"ContainerStarted","Data":"76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924"} Nov 03 07:00:48.384941 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:48.384792 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbd4d94c4-qgm6l" event={"ID":"82ff8095-af7a-446d-a4d6-a3da3d7cc02f","Type":"ContainerStarted","Data":"1f78d6f4e49de71a4c52946e14368c63e323b750d3e48464607d3cc25ca554e9"} Nov 03 07:00:48.507740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:48.507674 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6fbd4d94c4-qgm6l" podStartSLOduration=2.507662017 podStartE2EDuration="2.507662017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:00:48.506974113 +0000 UTC m=+103.699515868" watchObservedRunningTime="2026-11-03 07:00:48.507662017 +0000 UTC m=+103.700203766" Nov 03 07:00:57.317503 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:57.317470 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:57.318044 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:57.317524 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:57.321911 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:57.321894 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:57.418319 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:57.418293 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:00:57.600912 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:00:57.600830 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fc4ff94d4-vx66p"] Nov 03 07:01:08.772276 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:01:08.772244 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:01:22.620818 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.620699 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7fc4ff94d4-vx66p" podUID="add150f7-6577-4cfa-9eb3-f03c219d2886" containerName="console" containerID="cri-o://53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72" gracePeriod=15 Nov 03 07:01:22.865525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.865503 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 07:01:22.964292 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964246 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add150f7-6577-4cfa-9eb3-f03c219d2886-tmp\" (UniqueName: \"kubernetes.io/empty-dir/add150f7-6577-4cfa-9eb3-f03c219d2886-tmp\") pod \"add150f7-6577-4cfa-9eb3-f03c219d2886\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " Nov 03 07:01:22.964398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964310 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-oauth-config\") pod \"add150f7-6577-4cfa-9eb3-f03c219d2886\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " Nov 03 07:01:22.964398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964336 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-service-ca\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-service-ca\") pod \"add150f7-6577-4cfa-9eb3-f03c219d2886\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " Nov 03 07:01:22.964398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964355 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-trusted-ca-bundle\") pod \"add150f7-6577-4cfa-9eb3-f03c219d2886\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " Nov 03 07:01:22.964398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964374 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-oauth-serving-cert\") pod \"add150f7-6577-4cfa-9eb3-f03c219d2886\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " Nov 03 07:01:22.964398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964393 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-serving-cert\") pod \"add150f7-6577-4cfa-9eb3-f03c219d2886\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " Nov 03 07:01:22.964656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964428 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add150f7-6577-4cfa-9eb3-f03c219d2886-kube-api-access-tpzsn\" (UniqueName: \"kubernetes.io/projected/add150f7-6577-4cfa-9eb3-f03c219d2886-kube-api-access-tpzsn\") pod \"add150f7-6577-4cfa-9eb3-f03c219d2886\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " Nov 03 07:01:22.964656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964481 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add150f7-6577-4cfa-9eb3-f03c219d2886-tmp" pod "add150f7-6577-4cfa-9eb3-f03c219d2886" (UID: "add150f7-6577-4cfa-9eb3-f03c219d2886"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:01:22.964656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964539 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-console-config\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-console-config\") pod \"add150f7-6577-4cfa-9eb3-f03c219d2886\" (UID: \"add150f7-6577-4cfa-9eb3-f03c219d2886\") " Nov 03 07:01:22.964878 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964831 4409 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/add150f7-6577-4cfa-9eb3-f03c219d2886-tmp\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:01:22.964878 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964830 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-service-ca" pod "add150f7-6577-4cfa-9eb3-f03c219d2886" (UID: "add150f7-6577-4cfa-9eb3-f03c219d2886"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:01:22.965074 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964949 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-trusted-ca-bundle" pod "add150f7-6577-4cfa-9eb3-f03c219d2886" (UID: "add150f7-6577-4cfa-9eb3-f03c219d2886"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:01:22.965074 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.964849 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-oauth-serving-cert" pod "add150f7-6577-4cfa-9eb3-f03c219d2886" (UID: "add150f7-6577-4cfa-9eb3-f03c219d2886"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:01:22.965074 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.965031 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-console-config" pod "add150f7-6577-4cfa-9eb3-f03c219d2886" (UID: "add150f7-6577-4cfa-9eb3-f03c219d2886"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:01:22.966599 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.966573 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-oauth-config" pod "add150f7-6577-4cfa-9eb3-f03c219d2886" (UID: "add150f7-6577-4cfa-9eb3-f03c219d2886"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:01:22.966760 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.966737 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add150f7-6577-4cfa-9eb3-f03c219d2886-kube-api-access-tpzsn" pod "add150f7-6577-4cfa-9eb3-f03c219d2886" (UID: "add150f7-6577-4cfa-9eb3-f03c219d2886"). InnerVolumeSpecName "kube-api-access-tpzsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:01:22.966760 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:22.966736 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-serving-cert" pod "add150f7-6577-4cfa-9eb3-f03c219d2886" (UID: "add150f7-6577-4cfa-9eb3-f03c219d2886"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:01:23.066025 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.065999 4409 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-oauth-config\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:01:23.066025 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.066022 4409 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:01:23.066160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.066032 4409 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-trusted-ca-bundle\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:01:23.066160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.066041 4409 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-oauth-serving-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:01:23.066160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.066050 4409 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/add150f7-6577-4cfa-9eb3-f03c219d2886-console-serving-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:01:23.066160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.066058 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpzsn\" (UniqueName: \"kubernetes.io/projected/add150f7-6577-4cfa-9eb3-f03c219d2886-kube-api-access-tpzsn\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:01:23.066160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.066067 4409 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/add150f7-6577-4cfa-9eb3-f03c219d2886-console-config\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:01:23.508630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.508595 4409 generic.go:356] "Generic (PLEG): container finished" podID="add150f7-6577-4cfa-9eb3-f03c219d2886" containerID="53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72" exitCode=0 Nov 03 07:01:23.508820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.508639 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fc4ff94d4-vx66p" Nov 03 07:01:23.508820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.508668 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fc4ff94d4-vx66p" event={"ID":"add150f7-6577-4cfa-9eb3-f03c219d2886","Type":"ContainerDied","Data":"53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72"} Nov 03 07:01:23.508820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.508734 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fc4ff94d4-vx66p" event={"ID":"add150f7-6577-4cfa-9eb3-f03c219d2886","Type":"ContainerDied","Data":"2444d65a7f1db2b963c69f2fcec60e4f4f7f7eecd28d7c9c89c28836309da8d1"} Nov 03 07:01:23.508820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.508756 4409 scope.go:122] "RemoveContainer" containerID="53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72" Nov 03 07:01:23.517206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.517183 4409 scope.go:122] "RemoveContainer" containerID="53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72" Nov 03 07:01:23.517442 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:01:23.517424 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72\": container with ID starting with 53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72 not found: ID does not exist" containerID="53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72" Nov 03 07:01:23.517480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.517449 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72"} err="failed to get container status \"53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72\": rpc error: code = NotFound desc = could not find container \"53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72\": container with ID starting with 53f6ee680fa4056f062dfc2697ad3a1613fdbe10bdaf3beccf1d5218018e9b72 not found: ID does not exist" Nov 03 07:01:23.677073 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.677050 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fc4ff94d4-vx66p"] Nov 03 07:01:23.726324 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:23.726294 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fc4ff94d4-vx66p"] Nov 03 07:01:24.909805 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:01:24.909774 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add150f7-6577-4cfa-9eb3-f03c219d2886" path="/var/lib/kubelet/pods/add150f7-6577-4cfa-9eb3-f03c219d2886/volumes" Nov 03 07:01:38.772509 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:01:38.772458 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:02:08.771776 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:02:08.771744 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:02:15.074212 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.074180 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-b8c584585-xrftv"] Nov 03 07:02:15.074621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.074437 4409 cpu_manager.go:414] "containerMap: removing container" podUID="add150f7-6577-4cfa-9eb3-f03c219d2886" containerName="console" Nov 03 07:02:15.074621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.074447 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add150f7-6577-4cfa-9eb3-f03c219d2886" containerName="console" Nov 03 07:02:15.074621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.074500 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="add150f7-6577-4cfa-9eb3-f03c219d2886" containerName="console" Nov 03 07:02:15.078386 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.078371 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.125013 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.124989 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71063e1-249c-4728-8270-dfff3301f92c-console-config\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.125120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.125024 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71063e1-249c-4728-8270-dfff3301f92c-trusted-ca-bundle\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.125120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.125042 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71063e1-249c-4728-8270-dfff3301f92c-oauth-serving-cert\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.125120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.125069 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71063e1-249c-4728-8270-dfff3301f92c-console-serving-cert\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.125222 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.125156 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71063e1-249c-4728-8270-dfff3301f92c-console-oauth-config\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.125222 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.125172 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71063e1-249c-4728-8270-dfff3301f92c-service-ca\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.125222 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.125189 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b71063e1-249c-4728-8270-dfff3301f92c-tmp\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.125222 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.125213 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvxwf\" (UniqueName: \"kubernetes.io/projected/b71063e1-249c-4728-8270-dfff3301f92c-kube-api-access-qvxwf\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.180841 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.180812 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b8c584585-xrftv"] Nov 03 07:02:15.226308 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.226285 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b71063e1-249c-4728-8270-dfff3301f92c-tmp\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.226413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.226317 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvxwf\" (UniqueName: \"kubernetes.io/projected/b71063e1-249c-4728-8270-dfff3301f92c-kube-api-access-qvxwf\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.226413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.226343 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71063e1-249c-4728-8270-dfff3301f92c-console-config\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.226413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.226378 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71063e1-249c-4728-8270-dfff3301f92c-trusted-ca-bundle\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.226413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.226395 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71063e1-249c-4728-8270-dfff3301f92c-oauth-serving-cert\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.226618 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.226430 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71063e1-249c-4728-8270-dfff3301f92c-console-serving-cert\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.226618 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.226538 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71063e1-249c-4728-8270-dfff3301f92c-console-oauth-config\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.226618 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.226576 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71063e1-249c-4728-8270-dfff3301f92c-service-ca\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.226785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.226679 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b71063e1-249c-4728-8270-dfff3301f92c-tmp\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.227335 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.227265 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71063e1-249c-4728-8270-dfff3301f92c-trusted-ca-bundle\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.227335 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.227323 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71063e1-249c-4728-8270-dfff3301f92c-service-ca\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.227492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.227324 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71063e1-249c-4728-8270-dfff3301f92c-oauth-serving-cert\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.227492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.227477 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71063e1-249c-4728-8270-dfff3301f92c-console-config\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.229026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.228996 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71063e1-249c-4728-8270-dfff3301f92c-console-oauth-config\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.229183 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.229164 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71063e1-249c-4728-8270-dfff3301f92c-console-serving-cert\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.289288 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.289266 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvxwf\" (UniqueName: \"kubernetes.io/projected/b71063e1-249c-4728-8270-dfff3301f92c-kube-api-access-qvxwf\") pod \"console-b8c584585-xrftv\" (UID: \"b71063e1-249c-4728-8270-dfff3301f92c\") " pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.389847 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.389800 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:15.817846 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:15.817818 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b8c584585-xrftv"] Nov 03 07:02:15.820530 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:02:15.820502 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71063e1_249c_4728_8270_dfff3301f92c.slice/crio-078e9ac730e36a3f0877f46ab14ab3ed3208d8bfb630853679470a64b5026fba WatchSource:0}: Error finding container 078e9ac730e36a3f0877f46ab14ab3ed3208d8bfb630853679470a64b5026fba: Status 404 returned error can't find the container with id 078e9ac730e36a3f0877f46ab14ab3ed3208d8bfb630853679470a64b5026fba Nov 03 07:02:16.694132 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:16.694098 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c584585-xrftv" event={"ID":"b71063e1-249c-4728-8270-dfff3301f92c","Type":"ContainerStarted","Data":"bf38b1404f66e4c4b46bf9066e19efd211e25539e7596c218fe60131cac6a8ce"} Nov 03 07:02:16.694536 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:16.694145 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c584585-xrftv" event={"ID":"b71063e1-249c-4728-8270-dfff3301f92c","Type":"ContainerStarted","Data":"078e9ac730e36a3f0877f46ab14ab3ed3208d8bfb630853679470a64b5026fba"} Nov 03 07:02:16.868413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:16.868363 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-b8c584585-xrftv" podStartSLOduration=1.8683496069999999 podStartE2EDuration="1.868349607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:02:16.818478122 +0000 UTC m=+192.011019871" watchObservedRunningTime="2026-11-03 07:02:16.868349607 +0000 UTC m=+192.060891357" Nov 03 07:02:25.390253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:25.390211 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:25.390652 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:25.390281 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:25.395000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:25.394977 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:25.732133 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:25.732056 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b8c584585-xrftv" Nov 03 07:02:25.894901 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:25.894869 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbd4d94c4-qgm6l"] Nov 03 07:02:38.774183 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:02:38.774149 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:02:50.914317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:50.914230 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6fbd4d94c4-qgm6l" podUID="82ff8095-af7a-446d-a4d6-a3da3d7cc02f" containerName="console" containerID="cri-o://76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924" gracePeriod=15 Nov 03 07:02:51.154485 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.154461 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:02:51.291844 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.291769 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-kube-api-access-grxqk\" (UniqueName: \"kubernetes.io/projected/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-kube-api-access-grxqk\") pod \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " Nov 03 07:02:51.291844 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.291832 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-service-ca\") pod \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " Nov 03 07:02:51.292014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.291913 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-oauth-config\") pod \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " Nov 03 07:02:51.292014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.291947 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-oauth-serving-cert\") pod \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " Nov 03 07:02:51.292014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.291990 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-serving-cert\") pod \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " Nov 03 07:02:51.292174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.292025 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-config\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-config\") pod \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " Nov 03 07:02:51.292174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.292065 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-tmp\") pod \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " Nov 03 07:02:51.292174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.292097 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-trusted-ca-bundle\") pod \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\" (UID: \"82ff8095-af7a-446d-a4d6-a3da3d7cc02f\") " Nov 03 07:02:51.292366 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.292320 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-service-ca" pod "82ff8095-af7a-446d-a4d6-a3da3d7cc02f" (UID: "82ff8095-af7a-446d-a4d6-a3da3d7cc02f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:02:51.292570 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.292508 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-tmp" pod "82ff8095-af7a-446d-a4d6-a3da3d7cc02f" (UID: "82ff8095-af7a-446d-a4d6-a3da3d7cc02f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:02:51.292783 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.292639 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-trusted-ca-bundle" pod "82ff8095-af7a-446d-a4d6-a3da3d7cc02f" (UID: "82ff8095-af7a-446d-a4d6-a3da3d7cc02f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:02:51.292783 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.292633 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-config" pod "82ff8095-af7a-446d-a4d6-a3da3d7cc02f" (UID: "82ff8095-af7a-446d-a4d6-a3da3d7cc02f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:02:51.292783 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.292734 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-oauth-serving-cert" pod "82ff8095-af7a-446d-a4d6-a3da3d7cc02f" (UID: "82ff8095-af7a-446d-a4d6-a3da3d7cc02f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:02:51.294351 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.294326 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-oauth-config" pod "82ff8095-af7a-446d-a4d6-a3da3d7cc02f" (UID: "82ff8095-af7a-446d-a4d6-a3da3d7cc02f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:02:51.294793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.294769 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-kube-api-access-grxqk" pod "82ff8095-af7a-446d-a4d6-a3da3d7cc02f" (UID: "82ff8095-af7a-446d-a4d6-a3da3d7cc02f"). InnerVolumeSpecName "kube-api-access-grxqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:02:51.294987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.294968 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-serving-cert" pod "82ff8095-af7a-446d-a4d6-a3da3d7cc02f" (UID: "82ff8095-af7a-446d-a4d6-a3da3d7cc02f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:02:51.393368 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.393348 4409 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-serving-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:02:51.393368 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.393369 4409 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-config\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:02:51.393487 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.393379 4409 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-tmp\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:02:51.393487 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.393387 4409 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-trusted-ca-bundle\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:02:51.393487 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.393396 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grxqk\" (UniqueName: \"kubernetes.io/projected/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-kube-api-access-grxqk\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:02:51.393487 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.393405 4409 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:02:51.393487 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.393413 4409 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-console-oauth-config\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:02:51.393487 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.393422 4409 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82ff8095-af7a-446d-a4d6-a3da3d7cc02f-oauth-serving-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:02:51.816005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.815971 4409 generic.go:356] "Generic (PLEG): container finished" podID="82ff8095-af7a-446d-a4d6-a3da3d7cc02f" containerID="76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924" exitCode=0 Nov 03 07:02:51.816185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.816021 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbd4d94c4-qgm6l" Nov 03 07:02:51.816185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.816012 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbd4d94c4-qgm6l" event={"ID":"82ff8095-af7a-446d-a4d6-a3da3d7cc02f","Type":"ContainerDied","Data":"76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924"} Nov 03 07:02:51.816185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.816088 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbd4d94c4-qgm6l" event={"ID":"82ff8095-af7a-446d-a4d6-a3da3d7cc02f","Type":"ContainerDied","Data":"1f78d6f4e49de71a4c52946e14368c63e323b750d3e48464607d3cc25ca554e9"} Nov 03 07:02:51.816185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.816123 4409 scope.go:122] "RemoveContainer" containerID="76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924" Nov 03 07:02:51.824735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.824705 4409 scope.go:122] "RemoveContainer" containerID="76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924" Nov 03 07:02:51.825010 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:02:51.824993 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924\": container with ID starting with 76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924 not found: ID does not exist" containerID="76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924" Nov 03 07:02:51.825051 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.825019 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924"} err="failed to get container status \"76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924\": rpc error: code = NotFound desc = could not find container \"76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924\": container with ID starting with 76ba52cb4a3e7d5283d6d20043dc6634f4f41cffe14bd75040763a0ba4b03924 not found: ID does not exist" Nov 03 07:02:51.993881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:51.993850 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbd4d94c4-qgm6l"] Nov 03 07:02:52.043210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:52.043185 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fbd4d94c4-qgm6l"] Nov 03 07:02:52.910033 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:02:52.910004 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ff8095-af7a-446d-a4d6-a3da3d7cc02f" path="/var/lib/kubelet/pods/82ff8095-af7a-446d-a4d6-a3da3d7cc02f/volumes" Nov 03 07:03:08.772984 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:03:08.772944 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:03:38.773055 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:03:38.773009 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:04:04.915793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:04.915759 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:04:04.918171 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:04.918149 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:04:08.774180 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:04:08.774142 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:04:38.773968 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:04:38.773932 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:04:46.471887 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.471853 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6"] Nov 03 07:04:46.472288 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.472135 4409 cpu_manager.go:414] "containerMap: removing container" podUID="82ff8095-af7a-446d-a4d6-a3da3d7cc02f" containerName="console" Nov 03 07:04:46.472288 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.472145 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ff8095-af7a-446d-a4d6-a3da3d7cc02f" containerName="console" Nov 03 07:04:46.472288 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.472192 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ff8095-af7a-446d-a4d6-a3da3d7cc02f" containerName="console" Nov 03 07:04:46.475133 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.475116 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:04:46.578052 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.578022 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6\" (UID: \"f17547f3-f680-4458-93df-64dd02c31f8c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:04:46.578205 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.578058 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4m6z\" (UniqueName: \"kubernetes.io/projected/f17547f3-f680-4458-93df-64dd02c31f8c-kube-api-access-b4m6z\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6\" (UID: \"f17547f3-f680-4458-93df-64dd02c31f8c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:04:46.578205 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.578107 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6\" (UID: \"f17547f3-f680-4458-93df-64dd02c31f8c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:04:46.578336 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.578318 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-z2v9m\"" Nov 03 07:04:46.578413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.578397 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 07:04:46.578413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.578406 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 07:04:46.596506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.596483 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6"] Nov 03 07:04:46.603875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.603852 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b"] Nov 03 07:04:46.607101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.607084 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:04:46.664080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.664051 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-qg5wx\"" Nov 03 07:04:46.678489 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.678462 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b\" (UID: \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:04:46.678621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.678557 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6\" (UID: \"f17547f3-f680-4458-93df-64dd02c31f8c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:04:46.678621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.678590 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4m6z\" (UniqueName: \"kubernetes.io/projected/f17547f3-f680-4458-93df-64dd02c31f8c-kube-api-access-b4m6z\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6\" (UID: \"f17547f3-f680-4458-93df-64dd02c31f8c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:04:46.678761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.678650 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx7q5\" (UniqueName: \"kubernetes.io/projected/86be8fb7-0b05-411f-80d5-a0d86d5ad879-kube-api-access-kx7q5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b\" (UID: \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:04:46.678761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.678688 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b\" (UID: \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:04:46.678857 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.678797 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6\" (UID: \"f17547f3-f680-4458-93df-64dd02c31f8c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:04:46.679067 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.679046 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6\" (UID: \"f17547f3-f680-4458-93df-64dd02c31f8c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:04:46.679140 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.679089 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6\" (UID: \"f17547f3-f680-4458-93df-64dd02c31f8c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:04:46.709624 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.709595 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b"] Nov 03 07:04:46.753784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.753730 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4m6z\" (UniqueName: \"kubernetes.io/projected/f17547f3-f680-4458-93df-64dd02c31f8c-kube-api-access-b4m6z\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6\" (UID: \"f17547f3-f680-4458-93df-64dd02c31f8c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:04:46.779949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.779930 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b\" (UID: \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:04:46.780038 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.779977 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b\" (UID: \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:04:46.780084 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.780039 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx7q5\" (UniqueName: \"kubernetes.io/projected/86be8fb7-0b05-411f-80d5-a0d86d5ad879-kube-api-access-kx7q5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b\" (UID: \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:04:46.780332 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.780315 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b\" (UID: \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:04:46.780373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.780355 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b\" (UID: \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:04:46.789862 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.789840 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:04:46.844014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.843989 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx7q5\" (UniqueName: \"kubernetes.io/projected/86be8fb7-0b05-411f-80d5-a0d86d5ad879-kube-api-access-kx7q5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b\" (UID: \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:04:46.924227 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:46.924199 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:04:47.149837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.149810 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f"] Nov 03 07:04:47.153214 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.153198 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.165899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.165878 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6"] Nov 03 07:04:47.201314 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:04:47.201291 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17547f3_f680_4458_93df_64dd02c31f8c.slice/crio-96fe53605f4ca6fc21cb831a0f7981c6fef8422af1ada6e5e67773df80865f8e WatchSource:0}: Error finding container 96fe53605f4ca6fc21cb831a0f7981c6fef8422af1ada6e5e67773df80865f8e: Status 404 returned error can't find the container with id 96fe53605f4ca6fc21cb831a0f7981c6fef8422af1ada6e5e67773df80865f8e Nov 03 07:04:47.203059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.203044 4409 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:04:47.209931 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.209915 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 07:04:47.210119 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.210073 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 07:04:47.210215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.210200 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 07:04:47.210284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.210232 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 07:04:47.210284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.210260 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zrhnm\"" Nov 03 07:04:47.210284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.210280 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 07:04:47.210381 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.210290 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 07:04:47.223087 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.223068 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f"] Nov 03 07:04:47.284501 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.284473 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.284624 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.284510 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.284624 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.284551 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.284735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.284652 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.284735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.284677 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh7t8\" (UniqueName: \"kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-kube-api-access-dh7t8\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.284735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.284701 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.284864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.284813 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.298265 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.298246 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b"] Nov 03 07:04:47.300628 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:04:47.300608 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86be8fb7_0b05_411f_80d5_a0d86d5ad879.slice/crio-eb8530f6832b1e139afc7df8383b3e5019631853f415d2e0c48baefbc1fb0c8b WatchSource:0}: Error finding container eb8530f6832b1e139afc7df8383b3e5019631853f415d2e0c48baefbc1fb0c8b: Status 404 returned error can't find the container with id eb8530f6832b1e139afc7df8383b3e5019631853f415d2e0c48baefbc1fb0c8b Nov 03 07:04:47.385880 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.385856 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.385980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.385901 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.385980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.385943 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.385980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.385961 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.386136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.385999 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.386136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.386034 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.386136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.386059 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7t8\" (UniqueName: \"kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-kube-api-access-dh7t8\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.386666 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.386639 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.388369 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.388349 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.388526 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.388511 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.388562 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.388533 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.388599 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.388588 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.449041 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.448982 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh7t8\" (UniqueName: \"kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-kube-api-access-dh7t8\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.449041 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.449012 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gvv6f\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.464419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.464404 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:47.849118 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:47.849081 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f"] Nov 03 07:04:47.851936 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:04:47.851907 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e08d1a8_a40d_481c_b128_d1987d3d6cd1.slice/crio-457b133564a72f1d27c25151eab320886c9fac1fda17bdec5091281e28a869a3 WatchSource:0}: Error finding container 457b133564a72f1d27c25151eab320886c9fac1fda17bdec5091281e28a869a3: Status 404 returned error can't find the container with id 457b133564a72f1d27c25151eab320886c9fac1fda17bdec5091281e28a869a3 Nov 03 07:04:48.205270 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:48.205185 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" event={"ID":"0e08d1a8-a40d-481c-b128-d1987d3d6cd1","Type":"ContainerStarted","Data":"457b133564a72f1d27c25151eab320886c9fac1fda17bdec5091281e28a869a3"} Nov 03 07:04:48.206866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:48.206835 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" event={"ID":"f17547f3-f680-4458-93df-64dd02c31f8c","Type":"ContainerStarted","Data":"96fe53605f4ca6fc21cb831a0f7981c6fef8422af1ada6e5e67773df80865f8e"} Nov 03 07:04:48.208147 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:48.208110 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" event={"ID":"86be8fb7-0b05-411f-80d5-a0d86d5ad879","Type":"ContainerStarted","Data":"eb8530f6832b1e139afc7df8383b3e5019631853f415d2e0c48baefbc1fb0c8b"} Nov 03 07:04:52.406215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:52.406179 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:04:52.406565 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:52.406270 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:04:53.237661 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:53.236552 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" event={"ID":"0e08d1a8-a40d-481c-b128-d1987d3d6cd1","Type":"ContainerStarted","Data":"dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029"} Nov 03 07:04:53.237661 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:53.237600 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:53.239567 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:53.239449 4409 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-gvv6f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:04:53.239567 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:53.239536 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" podUID="0e08d1a8-a40d-481c-b128-d1987d3d6cd1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:04:53.449844 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:53.449780 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" podStartSLOduration=1.898743644 podStartE2EDuration="6.44975814s" totalImagesPullingTime="4.551014496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:04:53.409955593 +0000 UTC m=+348.602497350" watchObservedRunningTime="2026-11-03 07:04:53.44975814 +0000 UTC m=+348.642299885" Nov 03 07:04:54.242545 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:54.242511 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:04:55.246386 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:55.246349 4409 generic.go:356] "Generic (PLEG): container finished" podID="86be8fb7-0b05-411f-80d5-a0d86d5ad879" containerID="d8627de6df3294891bcc3f9c39be4abcebb00a5ab2c74fe2cd65271f7fb5fe77" exitCode=0 Nov 03 07:04:55.246809 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:55.246438 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" event={"ID":"86be8fb7-0b05-411f-80d5-a0d86d5ad879","Type":"ContainerDied","Data":"d8627de6df3294891bcc3f9c39be4abcebb00a5ab2c74fe2cd65271f7fb5fe77"} Nov 03 07:04:55.247952 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:55.247932 4409 generic.go:356] "Generic (PLEG): container finished" podID="f17547f3-f680-4458-93df-64dd02c31f8c" containerID="98da0fce61ca417689864cb0084e21179aa462f19cef3d587e35a213efdc013b" exitCode=0 Nov 03 07:04:55.248050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:55.248004 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" event={"ID":"f17547f3-f680-4458-93df-64dd02c31f8c","Type":"ContainerDied","Data":"98da0fce61ca417689864cb0084e21179aa462f19cef3d587e35a213efdc013b"} Nov 03 07:04:59.265865 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:59.265829 4409 generic.go:356] "Generic (PLEG): container finished" podID="f17547f3-f680-4458-93df-64dd02c31f8c" containerID="e348ada56605a3dfbc2d352b2ab429d9bd3bd5ff29c2fb7c4651748e99e6e570" exitCode=0 Nov 03 07:04:59.266358 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:59.265913 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" event={"ID":"f17547f3-f680-4458-93df-64dd02c31f8c","Type":"ContainerDied","Data":"e348ada56605a3dfbc2d352b2ab429d9bd3bd5ff29c2fb7c4651748e99e6e570"} Nov 03 07:04:59.267790 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:59.267770 4409 generic.go:356] "Generic (PLEG): container finished" podID="86be8fb7-0b05-411f-80d5-a0d86d5ad879" containerID="5b7d616f09cbaac5779aba79b0c065d168e145d4bdd9c6cc89b4e872dd769a34" exitCode=0 Nov 03 07:04:59.267865 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:04:59.267798 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" event={"ID":"86be8fb7-0b05-411f-80d5-a0d86d5ad879","Type":"ContainerDied","Data":"5b7d616f09cbaac5779aba79b0c065d168e145d4bdd9c6cc89b4e872dd769a34"} Nov 03 07:05:07.303931 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:07.303895 4409 generic.go:356] "Generic (PLEG): container finished" podID="f17547f3-f680-4458-93df-64dd02c31f8c" containerID="68141a0093b786b482a611a0f72c090c1c1a4e11bccc63ad63570243420fa204" exitCode=0 Nov 03 07:05:07.304340 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:07.304007 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" event={"ID":"f17547f3-f680-4458-93df-64dd02c31f8c","Type":"ContainerDied","Data":"68141a0093b786b482a611a0f72c090c1c1a4e11bccc63ad63570243420fa204"} Nov 03 07:05:07.306510 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:07.306487 4409 generic.go:356] "Generic (PLEG): container finished" podID="86be8fb7-0b05-411f-80d5-a0d86d5ad879" containerID="f281c6c3811b019321c63eb63c2552ad44ac5f31d46c51777df58bcadd1746d7" exitCode=0 Nov 03 07:05:07.306631 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:07.306539 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" event={"ID":"86be8fb7-0b05-411f-80d5-a0d86d5ad879","Type":"ContainerDied","Data":"f281c6c3811b019321c63eb63c2552ad44ac5f31d46c51777df58bcadd1746d7"} Nov 03 07:05:08.490641 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.490614 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:05:08.514753 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.514729 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:05:08.572456 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.572405 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-bundle\") pod \"f17547f3-f680-4458-93df-64dd02c31f8c\" (UID: \"f17547f3-f680-4458-93df-64dd02c31f8c\") " Nov 03 07:05:08.572456 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.572439 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-util\" (UniqueName: \"kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-util\") pod \"f17547f3-f680-4458-93df-64dd02c31f8c\" (UID: \"f17547f3-f680-4458-93df-64dd02c31f8c\") " Nov 03 07:05:08.572595 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.572462 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-util\" (UniqueName: \"kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-util\") pod \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\" (UID: \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\") " Nov 03 07:05:08.572595 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.572482 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86be8fb7-0b05-411f-80d5-a0d86d5ad879-kube-api-access-kx7q5\" (UniqueName: \"kubernetes.io/projected/86be8fb7-0b05-411f-80d5-a0d86d5ad879-kube-api-access-kx7q5\") pod \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\" (UID: \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\") " Nov 03 07:05:08.572595 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.572503 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-bundle\" (UniqueName: \"kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-bundle\") pod \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\" (UID: \"86be8fb7-0b05-411f-80d5-a0d86d5ad879\") " Nov 03 07:05:08.572595 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.572545 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17547f3-f680-4458-93df-64dd02c31f8c-kube-api-access-b4m6z\" (UniqueName: \"kubernetes.io/projected/f17547f3-f680-4458-93df-64dd02c31f8c-kube-api-access-b4m6z\") pod \"f17547f3-f680-4458-93df-64dd02c31f8c\" (UID: \"f17547f3-f680-4458-93df-64dd02c31f8c\") " Nov 03 07:05:08.572988 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.572946 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-bundle" pod "86be8fb7-0b05-411f-80d5-a0d86d5ad879" (UID: "86be8fb7-0b05-411f-80d5-a0d86d5ad879"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:08.573372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.573349 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-bundle" pod "f17547f3-f680-4458-93df-64dd02c31f8c" (UID: "f17547f3-f680-4458-93df-64dd02c31f8c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:08.575020 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.574995 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17547f3-f680-4458-93df-64dd02c31f8c-kube-api-access-b4m6z" pod "f17547f3-f680-4458-93df-64dd02c31f8c" (UID: "f17547f3-f680-4458-93df-64dd02c31f8c"). InnerVolumeSpecName "kube-api-access-b4m6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:05:08.575106 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.575013 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86be8fb7-0b05-411f-80d5-a0d86d5ad879-kube-api-access-kx7q5" pod "86be8fb7-0b05-411f-80d5-a0d86d5ad879" (UID: "86be8fb7-0b05-411f-80d5-a0d86d5ad879"). InnerVolumeSpecName "kube-api-access-kx7q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:05:08.579034 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.579010 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-util" pod "86be8fb7-0b05-411f-80d5-a0d86d5ad879" (UID: "86be8fb7-0b05-411f-80d5-a0d86d5ad879"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:08.579308 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.579292 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-util" pod "f17547f3-f680-4458-93df-64dd02c31f8c" (UID: "f17547f3-f680-4458-93df-64dd02c31f8c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:05:08.673098 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.673074 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4m6z\" (UniqueName: \"kubernetes.io/projected/f17547f3-f680-4458-93df-64dd02c31f8c-kube-api-access-b4m6z\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:08.673098 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.673096 4409 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-bundle\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:08.673226 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.673105 4409 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f17547f3-f680-4458-93df-64dd02c31f8c-util\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:08.673226 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.673113 4409 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-util\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:08.673226 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.673121 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx7q5\" (UniqueName: \"kubernetes.io/projected/86be8fb7-0b05-411f-80d5-a0d86d5ad879-kube-api-access-kx7q5\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:08.673226 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:08.673130 4409 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86be8fb7-0b05-411f-80d5-a0d86d5ad879-bundle\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:05:08.771607 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:05:08.771576 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:05:09.315591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:09.315565 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" Nov 03 07:05:09.315591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:09.315582 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cp82b6" event={"ID":"f17547f3-f680-4458-93df-64dd02c31f8c","Type":"ContainerDied","Data":"96fe53605f4ca6fc21cb831a0f7981c6fef8422af1ada6e5e67773df80865f8e"} Nov 03 07:05:09.315853 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:09.315629 4409 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96fe53605f4ca6fc21cb831a0f7981c6fef8422af1ada6e5e67773df80865f8e" Nov 03 07:05:09.317928 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:09.317904 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" event={"ID":"86be8fb7-0b05-411f-80d5-a0d86d5ad879","Type":"ContainerDied","Data":"eb8530f6832b1e139afc7df8383b3e5019631853f415d2e0c48baefbc1fb0c8b"} Nov 03 07:05:09.318020 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:09.317947 4409 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb8530f6832b1e139afc7df8383b3e5019631853f415d2e0c48baefbc1fb0c8b" Nov 03 07:05:09.318020 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:09.317968 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445885b" Nov 03 07:05:29.458381 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458347 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg"] Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458756 4409 cpu_manager.go:414] "containerMap: removing container" podUID="86be8fb7-0b05-411f-80d5-a0d86d5ad879" containerName="util" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458774 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86be8fb7-0b05-411f-80d5-a0d86d5ad879" containerName="util" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458798 4409 cpu_manager.go:414] "containerMap: removing container" podUID="f17547f3-f680-4458-93df-64dd02c31f8c" containerName="pull" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458806 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17547f3-f680-4458-93df-64dd02c31f8c" containerName="pull" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458817 4409 cpu_manager.go:414] "containerMap: removing container" podUID="f17547f3-f680-4458-93df-64dd02c31f8c" containerName="extract" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458825 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17547f3-f680-4458-93df-64dd02c31f8c" containerName="extract" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458841 4409 cpu_manager.go:414] "containerMap: removing container" podUID="86be8fb7-0b05-411f-80d5-a0d86d5ad879" containerName="pull" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458848 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86be8fb7-0b05-411f-80d5-a0d86d5ad879" containerName="pull" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458865 4409 cpu_manager.go:414] "containerMap: removing container" podUID="86be8fb7-0b05-411f-80d5-a0d86d5ad879" containerName="extract" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458872 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86be8fb7-0b05-411f-80d5-a0d86d5ad879" containerName="extract" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458886 4409 cpu_manager.go:414] "containerMap: removing container" podUID="f17547f3-f680-4458-93df-64dd02c31f8c" containerName="util" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458893 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17547f3-f680-4458-93df-64dd02c31f8c" containerName="util" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458968 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="86be8fb7-0b05-411f-80d5-a0d86d5ad879" containerName="extract" Nov 03 07:05:29.459012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.458985 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17547f3-f680-4458-93df-64dd02c31f8c" containerName="extract" Nov 03 07:05:29.461107 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.461086 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.507777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.507745 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.507904 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.507793 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-manifests\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.507904 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.507879 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7czx\" (UniqueName: \"kubernetes.io/projected/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-kube-api-access-c7czx\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.507997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.507907 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-metrics-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.507997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.507924 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-webhook-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.508097 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.508045 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-tmp\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.517223 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.517203 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 07:05:29.517437 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.517420 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 07:05:29.565378 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.565359 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 07:05:29.565509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.565467 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 07:05:29.565623 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.565607 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-frrnv\"" Nov 03 07:05:29.565623 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.565617 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 07:05:29.567308 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.567292 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg"] Nov 03 07:05:29.608861 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.608834 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7czx\" (UniqueName: \"kubernetes.io/projected/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-kube-api-access-c7czx\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.608964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.608864 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-metrics-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.608964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.608883 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-webhook-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.608964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.608911 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-tmp\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.608964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.608950 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.609151 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.608990 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-manifests\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.609461 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.609439 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-manifests\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.611389 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.611367 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-tmp\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.611573 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.611550 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.611914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.611895 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-metrics-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.612003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.611963 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-webhook-cert\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.671735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.671688 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7czx\" (UniqueName: \"kubernetes.io/projected/1fcb520a-21a6-4ed0-b0e5-edbd35932d8d-kube-api-access-c7czx\") pod \"opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg\" (UID: \"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:29.773870 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:29.773817 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:30.263828 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:30.263804 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg"] Nov 03 07:05:30.266233 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:05:30.266207 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcb520a_21a6_4ed0_b0e5_edbd35932d8d.slice/crio-7a0de904d130dc102284875696fd091028ba53f44249670e047da5d51a88bdb0 WatchSource:0}: Error finding container 7a0de904d130dc102284875696fd091028ba53f44249670e047da5d51a88bdb0: Status 404 returned error can't find the container with id 7a0de904d130dc102284875696fd091028ba53f44249670e047da5d51a88bdb0 Nov 03 07:05:30.391847 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:30.391814 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" event={"ID":"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d","Type":"ContainerStarted","Data":"7a0de904d130dc102284875696fd091028ba53f44249670e047da5d51a88bdb0"} Nov 03 07:05:34.409198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:34.409163 4409 generic.go:356] "Generic (PLEG): container finished" podID="1fcb520a-21a6-4ed0-b0e5-edbd35932d8d" containerID="80b23155b066ae17c134b521754d1e1cc4554f7d020ddd51d77c7edf3a5ce219" exitCode=0 Nov 03 07:05:34.409532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:34.409250 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" event={"ID":"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d","Type":"ContainerDied","Data":"80b23155b066ae17c134b521754d1e1cc4554f7d020ddd51d77c7edf3a5ce219"} Nov 03 07:05:35.415058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:35.414969 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" event={"ID":"1fcb520a-21a6-4ed0-b0e5-edbd35932d8d","Type":"ContainerStarted","Data":"3894865d027fded4c6160d83ab4dd03b33a2c7289bbd862d93cc42537e5abf76"} Nov 03 07:05:35.415426 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:35.415063 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:05:35.660230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:35.660175 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" podStartSLOduration=1.969539049 podStartE2EDuration="6.660156159s" totalImagesPullingTime="4.28396711s" totalInitContainerRuntime="406.65ms" isStatefulPod=false podCreationTimestamp="2026-11-03 07:05:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:05:35.610188829 +0000 UTC m=+390.802730580" watchObservedRunningTime="2026-11-03 07:05:35.660156159 +0000 UTC m=+390.852697980" Nov 03 07:05:38.773209 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:05:38.773170 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:05:46.421177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:05:46.421072 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69dc8cbf8f-4tsmg" Nov 03 07:06:08.772823 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:06:08.772787 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:06:09.394949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.392777 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86b9d545b-j69hh"] Nov 03 07:06:09.399938 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.399912 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" Nov 03 07:06:09.503830 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.503802 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86b9d545b-j69hh"] Nov 03 07:06:09.521830 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.521805 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d8f88ce-ff40-439d-ba00-1881218b47f6-tmp\") pod \"kube-auth-proxy-86b9d545b-j69hh\" (UID: \"5d8f88ce-ff40-439d-ba00-1881218b47f6\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" Nov 03 07:06:09.521965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.521836 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d8f88ce-ff40-439d-ba00-1881218b47f6-tls-certs\") pod \"kube-auth-proxy-86b9d545b-j69hh\" (UID: \"5d8f88ce-ff40-439d-ba00-1881218b47f6\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" Nov 03 07:06:09.521965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.521856 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqx6n\" (UniqueName: \"kubernetes.io/projected/5d8f88ce-ff40-439d-ba00-1881218b47f6-kube-api-access-hqx6n\") pod \"kube-auth-proxy-86b9d545b-j69hh\" (UID: \"5d8f88ce-ff40-439d-ba00-1881218b47f6\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" Nov 03 07:06:09.567933 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.567905 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 07:06:09.567933 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.567915 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-skt9p\"" Nov 03 07:06:09.568100 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.567921 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 07:06:09.622475 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.622447 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d8f88ce-ff40-439d-ba00-1881218b47f6-tmp\") pod \"kube-auth-proxy-86b9d545b-j69hh\" (UID: \"5d8f88ce-ff40-439d-ba00-1881218b47f6\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" Nov 03 07:06:09.622658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.622483 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d8f88ce-ff40-439d-ba00-1881218b47f6-tls-certs\") pod \"kube-auth-proxy-86b9d545b-j69hh\" (UID: \"5d8f88ce-ff40-439d-ba00-1881218b47f6\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" Nov 03 07:06:09.622658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.622500 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqx6n\" (UniqueName: \"kubernetes.io/projected/5d8f88ce-ff40-439d-ba00-1881218b47f6-kube-api-access-hqx6n\") pod \"kube-auth-proxy-86b9d545b-j69hh\" (UID: \"5d8f88ce-ff40-439d-ba00-1881218b47f6\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" Nov 03 07:06:09.625295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.625269 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d8f88ce-ff40-439d-ba00-1881218b47f6-tmp\") pod \"kube-auth-proxy-86b9d545b-j69hh\" (UID: \"5d8f88ce-ff40-439d-ba00-1881218b47f6\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" Nov 03 07:06:09.625522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.625503 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d8f88ce-ff40-439d-ba00-1881218b47f6-tls-certs\") pod \"kube-auth-proxy-86b9d545b-j69hh\" (UID: \"5d8f88ce-ff40-439d-ba00-1881218b47f6\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" Nov 03 07:06:09.686275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.686208 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqx6n\" (UniqueName: \"kubernetes.io/projected/5d8f88ce-ff40-439d-ba00-1881218b47f6-kube-api-access-hqx6n\") pod \"kube-auth-proxy-86b9d545b-j69hh\" (UID: \"5d8f88ce-ff40-439d-ba00-1881218b47f6\") " pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" Nov 03 07:06:09.710043 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:09.710019 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" Nov 03 07:06:10.087347 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:10.087322 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86b9d545b-j69hh"] Nov 03 07:06:10.089953 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:06:10.089922 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8f88ce_ff40_439d_ba00_1881218b47f6.slice/crio-40549115aaa0c0f5d41fc68715c7788e80b9b0ee4fc5d3835cfa3e0f88d7fda1 WatchSource:0}: Error finding container 40549115aaa0c0f5d41fc68715c7788e80b9b0ee4fc5d3835cfa3e0f88d7fda1: Status 404 returned error can't find the container with id 40549115aaa0c0f5d41fc68715c7788e80b9b0ee4fc5d3835cfa3e0f88d7fda1 Nov 03 07:06:10.557799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:10.557772 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" event={"ID":"5d8f88ce-ff40-439d-ba00-1881218b47f6","Type":"ContainerStarted","Data":"40549115aaa0c0f5d41fc68715c7788e80b9b0ee4fc5d3835cfa3e0f88d7fda1"} Nov 03 07:06:13.573228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:13.573189 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" event={"ID":"5d8f88ce-ff40-439d-ba00-1881218b47f6","Type":"ContainerStarted","Data":"26e1eba96ba092e1d69e6e9eabf509354b4d0a7072f7746827a21a2e753dd66c"} Nov 03 07:06:13.695410 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:13.695358 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86b9d545b-j69hh" podStartSLOduration=1.802911409 podStartE2EDuration="4.695341879s" totalImagesPullingTime="2.89243047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:06:13.694522069 +0000 UTC m=+428.887063821" watchObservedRunningTime="2026-11-03 07:06:13.695341879 +0000 UTC m=+428.887883628" Nov 03 07:06:17.770475 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:17.770442 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9jz7x"] Nov 03 07:06:17.773519 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:17.773504 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" Nov 03 07:06:17.828466 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:17.828438 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 07:06:17.828570 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:17.828469 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 07:06:17.877293 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:17.877261 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 07:06:17.877419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:17.877357 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t675q\"" Nov 03 07:06:17.879298 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:17.879278 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9jz7x"] Nov 03 07:06:17.888333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:17.888309 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cg4z\" (UniqueName: \"kubernetes.io/projected/448c0874-d9d9-47e2-934f-ffa816993346-kube-api-access-8cg4z\") pod \"limitador-limitador-74dfcfb894-9jz7x\" (UID: \"448c0874-d9d9-47e2-934f-ffa816993346\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" Nov 03 07:06:17.888494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:17.888471 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/448c0874-d9d9-47e2-934f-ffa816993346-config-file\") pod \"limitador-limitador-74dfcfb894-9jz7x\" (UID: \"448c0874-d9d9-47e2-934f-ffa816993346\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" Nov 03 07:06:17.989281 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:17.989249 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/448c0874-d9d9-47e2-934f-ffa816993346-config-file\") pod \"limitador-limitador-74dfcfb894-9jz7x\" (UID: \"448c0874-d9d9-47e2-934f-ffa816993346\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" Nov 03 07:06:17.989421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:17.989290 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cg4z\" (UniqueName: \"kubernetes.io/projected/448c0874-d9d9-47e2-934f-ffa816993346-kube-api-access-8cg4z\") pod \"limitador-limitador-74dfcfb894-9jz7x\" (UID: \"448c0874-d9d9-47e2-934f-ffa816993346\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" Nov 03 07:06:17.989898 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:17.989880 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/448c0874-d9d9-47e2-934f-ffa816993346-config-file\") pod \"limitador-limitador-74dfcfb894-9jz7x\" (UID: \"448c0874-d9d9-47e2-934f-ffa816993346\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" Nov 03 07:06:18.013962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:18.013930 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9jz7x"] Nov 03 07:06:18.052256 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:18.052238 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cg4z\" (UniqueName: \"kubernetes.io/projected/448c0874-d9d9-47e2-934f-ffa816993346-kube-api-access-8cg4z\") pod \"limitador-limitador-74dfcfb894-9jz7x\" (UID: \"448c0874-d9d9-47e2-934f-ffa816993346\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" Nov 03 07:06:18.085780 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:18.085751 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" Nov 03 07:06:18.461960 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:18.461939 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9jz7x"] Nov 03 07:06:18.463902 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:06:18.463873 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448c0874_d9d9_47e2_934f_ffa816993346.slice/crio-2345ddee1a02d49101a62787759b2646d98a0badf3dd38149a19c095b6f17b7e WatchSource:0}: Error finding container 2345ddee1a02d49101a62787759b2646d98a0badf3dd38149a19c095b6f17b7e: Status 404 returned error can't find the container with id 2345ddee1a02d49101a62787759b2646d98a0badf3dd38149a19c095b6f17b7e Nov 03 07:06:18.593601 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:18.593570 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" event={"ID":"448c0874-d9d9-47e2-934f-ffa816993346","Type":"ContainerStarted","Data":"2345ddee1a02d49101a62787759b2646d98a0badf3dd38149a19c095b6f17b7e"} Nov 03 07:06:22.084990 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.084964 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mnd8x"] Nov 03 07:06:22.088149 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.088133 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" Nov 03 07:06:22.124654 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.124637 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc722\" (UniqueName: \"kubernetes.io/projected/fabbdc26-a95a-401a-b787-023b23fd24e7-kube-api-access-hc722\") pod \"authorino-56c6c9ddd5-mnd8x\" (UID: \"fabbdc26-a95a-401a-b787-023b23fd24e7\") " pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" Nov 03 07:06:22.145218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.145195 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mnd8x"] Nov 03 07:06:22.191985 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.191967 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zmffl\"" Nov 03 07:06:22.225119 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.225093 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc722\" (UniqueName: \"kubernetes.io/projected/fabbdc26-a95a-401a-b787-023b23fd24e7-kube-api-access-hc722\") pod \"authorino-56c6c9ddd5-mnd8x\" (UID: \"fabbdc26-a95a-401a-b787-023b23fd24e7\") " pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" Nov 03 07:06:22.288007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.287984 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc722\" (UniqueName: \"kubernetes.io/projected/fabbdc26-a95a-401a-b787-023b23fd24e7-kube-api-access-hc722\") pod \"authorino-56c6c9ddd5-mnd8x\" (UID: \"fabbdc26-a95a-401a-b787-023b23fd24e7\") " pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" Nov 03 07:06:22.412440 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.412380 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" Nov 03 07:06:22.612790 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.612761 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" event={"ID":"448c0874-d9d9-47e2-934f-ffa816993346","Type":"ContainerStarted","Data":"e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c"} Nov 03 07:06:22.612953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.612828 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" Nov 03 07:06:22.731682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.731589 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" podStartSLOduration=2.46263814 podStartE2EDuration="5.731572907s" totalImagesPullingTime="3.268934767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:06:22.731168155 +0000 UTC m=+437.923709906" watchObservedRunningTime="2026-11-03 07:06:22.731572907 +0000 UTC m=+437.924114657" Nov 03 07:06:22.844920 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.844893 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mnd8x"] Nov 03 07:06:22.848771 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:06:22.848745 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfabbdc26_a95a_401a_b787_023b23fd24e7.slice/crio-c87c093bedc42befbfbb4782793fce20452eed129bdc019eb2eae452b32babfa WatchSource:0}: Error finding container c87c093bedc42befbfbb4782793fce20452eed129bdc019eb2eae452b32babfa: Status 404 returned error can't find the container with id c87c093bedc42befbfbb4782793fce20452eed129bdc019eb2eae452b32babfa Nov 03 07:06:22.998542 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:22.998483 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q"] Nov 03 07:06:23.002947 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.002929 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.031722 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.031693 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f552b885-fd66-4001-a396-326521f15680-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-76r6q\" (UID: \"f552b885-fd66-4001-a396-326521f15680\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.031825 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.031747 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btkh7\" (UniqueName: \"kubernetes.io/projected/f552b885-fd66-4001-a396-326521f15680-kube-api-access-btkh7\") pod \"kserve-module-controller-manager-7566cf5fc8-76r6q\" (UID: \"f552b885-fd66-4001-a396-326521f15680\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.031901 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.031819 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f552b885-fd66-4001-a396-326521f15680-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-76r6q\" (UID: \"f552b885-fd66-4001-a396-326521f15680\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.031901 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.031886 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f552b885-fd66-4001-a396-326521f15680-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-76r6q\" (UID: \"f552b885-fd66-4001-a396-326521f15680\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.057318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.057298 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 07:06:23.057434 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.057420 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 07:06:23.057624 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.057612 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-gdczv\"" Nov 03 07:06:23.133204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.133177 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btkh7\" (UniqueName: \"kubernetes.io/projected/f552b885-fd66-4001-a396-326521f15680-kube-api-access-btkh7\") pod \"kserve-module-controller-manager-7566cf5fc8-76r6q\" (UID: \"f552b885-fd66-4001-a396-326521f15680\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.133615 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.133211 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f552b885-fd66-4001-a396-326521f15680-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-76r6q\" (UID: \"f552b885-fd66-4001-a396-326521f15680\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.133615 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.133246 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f552b885-fd66-4001-a396-326521f15680-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-76r6q\" (UID: \"f552b885-fd66-4001-a396-326521f15680\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.133615 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.133301 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f552b885-fd66-4001-a396-326521f15680-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-76r6q\" (UID: \"f552b885-fd66-4001-a396-326521f15680\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.133615 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.133564 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f552b885-fd66-4001-a396-326521f15680-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-76r6q\" (UID: \"f552b885-fd66-4001-a396-326521f15680\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.133837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.133664 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f552b885-fd66-4001-a396-326521f15680-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-76r6q\" (UID: \"f552b885-fd66-4001-a396-326521f15680\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.157077 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.157056 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 07:06:23.158559 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.158542 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q"] Nov 03 07:06:23.165081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.165064 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f552b885-fd66-4001-a396-326521f15680-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-76r6q\" (UID: \"f552b885-fd66-4001-a396-326521f15680\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.196154 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.196135 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btkh7\" (UniqueName: \"kubernetes.io/projected/f552b885-fd66-4001-a396-326521f15680-kube-api-access-btkh7\") pod \"kserve-module-controller-manager-7566cf5fc8-76r6q\" (UID: \"f552b885-fd66-4001-a396-326521f15680\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.314867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.314833 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:23.618305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.618236 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" event={"ID":"fabbdc26-a95a-401a-b787-023b23fd24e7","Type":"ContainerStarted","Data":"c87c093bedc42befbfbb4782793fce20452eed129bdc019eb2eae452b32babfa"} Nov 03 07:06:23.737749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:23.737706 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q"] Nov 03 07:06:23.740492 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:06:23.740468 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf552b885_fd66_4001_a396_326521f15680.slice/crio-3d1f1663b58e180b0f2c2f96115c931e031a09b9a8a5d526ce732c746c49540d WatchSource:0}: Error finding container 3d1f1663b58e180b0f2c2f96115c931e031a09b9a8a5d526ce732c746c49540d: Status 404 returned error can't find the container with id 3d1f1663b58e180b0f2c2f96115c931e031a09b9a8a5d526ce732c746c49540d Nov 03 07:06:24.623793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:24.623756 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" event={"ID":"f552b885-fd66-4001-a396-326521f15680","Type":"ContainerStarted","Data":"3d1f1663b58e180b0f2c2f96115c931e031a09b9a8a5d526ce732c746c49540d"} Nov 03 07:06:26.633924 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:26.633883 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" event={"ID":"fabbdc26-a95a-401a-b787-023b23fd24e7","Type":"ContainerStarted","Data":"1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165"} Nov 03 07:06:26.635459 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:26.635435 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" event={"ID":"f552b885-fd66-4001-a396-326521f15680","Type":"ContainerStarted","Data":"78785c9d6360082d8e52684c74b3468968c54c3873ea0c0c787277965dfce27c"} Nov 03 07:06:26.635595 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:26.635562 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:26.754348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:26.754299 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" podStartSLOduration=1.967799591 podStartE2EDuration="4.754287057s" totalImagesPullingTime="2.786487466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:06:26.753405372 +0000 UTC m=+441.945947123" watchObservedRunningTime="2026-11-03 07:06:26.754287057 +0000 UTC m=+441.946828807" Nov 03 07:06:26.873905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:26.873859 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" podStartSLOduration=2.665269582 podStartE2EDuration="4.873841338s" totalImagesPullingTime="2.208571756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:06:26.872823538 +0000 UTC m=+442.065365288" watchObservedRunningTime="2026-11-03 07:06:26.873841338 +0000 UTC m=+442.066383084" Nov 03 07:06:33.619138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:33.619109 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" Nov 03 07:06:36.526330 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:36.526291 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9jz7x"] Nov 03 07:06:36.526764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:36.526463 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" podUID="448c0874-d9d9-47e2-934f-ffa816993346" containerName="limitador" containerID="cri-o://e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c" gracePeriod=30 Nov 03 07:06:37.065274 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.065251 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" Nov 03 07:06:37.154908 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.154848 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448c0874-d9d9-47e2-934f-ffa816993346-config-file\" (UniqueName: \"kubernetes.io/configmap/448c0874-d9d9-47e2-934f-ffa816993346-config-file\") pod \"448c0874-d9d9-47e2-934f-ffa816993346\" (UID: \"448c0874-d9d9-47e2-934f-ffa816993346\") " Nov 03 07:06:37.154908 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.154891 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448c0874-d9d9-47e2-934f-ffa816993346-kube-api-access-8cg4z\" (UniqueName: \"kubernetes.io/projected/448c0874-d9d9-47e2-934f-ffa816993346-kube-api-access-8cg4z\") pod \"448c0874-d9d9-47e2-934f-ffa816993346\" (UID: \"448c0874-d9d9-47e2-934f-ffa816993346\") " Nov 03 07:06:37.155169 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.155147 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448c0874-d9d9-47e2-934f-ffa816993346-config-file" pod "448c0874-d9d9-47e2-934f-ffa816993346" (UID: "448c0874-d9d9-47e2-934f-ffa816993346"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:06:37.157133 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.157111 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448c0874-d9d9-47e2-934f-ffa816993346-kube-api-access-8cg4z" pod "448c0874-d9d9-47e2-934f-ffa816993346" (UID: "448c0874-d9d9-47e2-934f-ffa816993346"). InnerVolumeSpecName "kube-api-access-8cg4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:06:37.255676 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.255643 4409 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/448c0874-d9d9-47e2-934f-ffa816993346-config-file\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:06:37.255676 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.255671 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cg4z\" (UniqueName: \"kubernetes.io/projected/448c0874-d9d9-47e2-934f-ffa816993346-kube-api-access-8cg4z\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:06:37.678945 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.678914 4409 generic.go:356] "Generic (PLEG): container finished" podID="448c0874-d9d9-47e2-934f-ffa816993346" containerID="e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c" exitCode=0 Nov 03 07:06:37.679429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.679251 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" Nov 03 07:06:37.679429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.679315 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" event={"ID":"448c0874-d9d9-47e2-934f-ffa816993346","Type":"ContainerDied","Data":"e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c"} Nov 03 07:06:37.679429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.679386 4409 scope.go:122] "RemoveContainer" containerID="e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c" Nov 03 07:06:37.679586 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.679422 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9jz7x" event={"ID":"448c0874-d9d9-47e2-934f-ffa816993346","Type":"ContainerDied","Data":"2345ddee1a02d49101a62787759b2646d98a0badf3dd38149a19c095b6f17b7e"} Nov 03 07:06:37.691271 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.691252 4409 scope.go:122] "RemoveContainer" containerID="e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c" Nov 03 07:06:37.691560 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:06:37.691537 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c\": container with ID starting with e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c not found: ID does not exist" containerID="e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c" Nov 03 07:06:37.691656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.691566 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c"} err="failed to get container status \"e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c\": rpc error: code = NotFound desc = could not find container \"e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c\": container with ID starting with e025dc98c41264ece3b50841adcc52473077b3f1e78db83210c609deb5d8af2c not found: ID does not exist" Nov 03 07:06:37.846729 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.846674 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9jz7x"] Nov 03 07:06:37.896198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:37.896166 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9jz7x"] Nov 03 07:06:38.773675 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:06:38.773643 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:06:38.911109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:38.911074 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448c0874-d9d9-47e2-934f-ffa816993346" path="/var/lib/kubelet/pods/448c0874-d9d9-47e2-934f-ffa816993346/volumes" Nov 03 07:06:40.698098 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.698061 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc"] Nov 03 07:06:40.698693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.698542 4409 cpu_manager.go:414] "containerMap: removing container" podUID="448c0874-d9d9-47e2-934f-ffa816993346" containerName="limitador" Nov 03 07:06:40.698693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.698560 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448c0874-d9d9-47e2-934f-ffa816993346" containerName="limitador" Nov 03 07:06:40.698693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.698645 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="448c0874-d9d9-47e2-934f-ffa816993346" containerName="limitador" Nov 03 07:06:40.704604 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.704580 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" Nov 03 07:06:40.760315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.760285 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc"] Nov 03 07:06:40.765211 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.765186 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 07:06:40.788452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.788425 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42h7k\" (UniqueName: \"kubernetes.io/projected/4887bd60-5f24-41a7-814c-9a84fadb3126-kube-api-access-42h7k\") pod \"postgres-7cdbf6dc44-hgbzc\" (UID: \"4887bd60-5f24-41a7-814c-9a84fadb3126\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" Nov 03 07:06:40.788605 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.788463 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4887bd60-5f24-41a7-814c-9a84fadb3126-data\") pod \"postgres-7cdbf6dc44-hgbzc\" (UID: \"4887bd60-5f24-41a7-814c-9a84fadb3126\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" Nov 03 07:06:40.808423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.808388 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 07:06:40.808423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.808399 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-c9ff9\"" Nov 03 07:06:40.808618 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.808450 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 07:06:40.889045 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.889005 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42h7k\" (UniqueName: \"kubernetes.io/projected/4887bd60-5f24-41a7-814c-9a84fadb3126-kube-api-access-42h7k\") pod \"postgres-7cdbf6dc44-hgbzc\" (UID: \"4887bd60-5f24-41a7-814c-9a84fadb3126\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" Nov 03 07:06:40.889239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.889067 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4887bd60-5f24-41a7-814c-9a84fadb3126-data\") pod \"postgres-7cdbf6dc44-hgbzc\" (UID: \"4887bd60-5f24-41a7-814c-9a84fadb3126\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" Nov 03 07:06:40.889508 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.889484 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4887bd60-5f24-41a7-814c-9a84fadb3126-data\") pod \"postgres-7cdbf6dc44-hgbzc\" (UID: \"4887bd60-5f24-41a7-814c-9a84fadb3126\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" Nov 03 07:06:40.956157 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:40.956087 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42h7k\" (UniqueName: \"kubernetes.io/projected/4887bd60-5f24-41a7-814c-9a84fadb3126-kube-api-access-42h7k\") pod \"postgres-7cdbf6dc44-hgbzc\" (UID: \"4887bd60-5f24-41a7-814c-9a84fadb3126\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" Nov 03 07:06:41.023615 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:41.023575 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" Nov 03 07:06:41.398744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:41.398609 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc"] Nov 03 07:06:41.401367 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:06:41.401325 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4887bd60_5f24_41a7_814c_9a84fadb3126.slice/crio-aba2262c93ae35640cee02f2be0884ce212770ebbf577dc0c1be83c054951367 WatchSource:0}: Error finding container aba2262c93ae35640cee02f2be0884ce212770ebbf577dc0c1be83c054951367: Status 404 returned error can't find the container with id aba2262c93ae35640cee02f2be0884ce212770ebbf577dc0c1be83c054951367 Nov 03 07:06:41.696383 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:41.696299 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" event={"ID":"4887bd60-5f24-41a7-814c-9a84fadb3126","Type":"ContainerStarted","Data":"aba2262c93ae35640cee02f2be0884ce212770ebbf577dc0c1be83c054951367"} Nov 03 07:06:47.640938 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:47.640900 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-76r6q" Nov 03 07:06:48.728036 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:48.727965 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" event={"ID":"4887bd60-5f24-41a7-814c-9a84fadb3126","Type":"ContainerStarted","Data":"4959eacf052e1446c3bf408597e486ee15e833e471892232e116ec1f476e04b9"} Nov 03 07:06:48.728371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:48.728102 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" Nov 03 07:06:48.851450 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:48.851400 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" podStartSLOduration=1.807979538 podStartE2EDuration="8.851387429s" totalImagesPullingTime="7.043407891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:06:48.851284846 +0000 UTC m=+464.043826598" watchObservedRunningTime="2026-11-03 07:06:48.851387429 +0000 UTC m=+464.043929179" Nov 03 07:06:54.761440 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:06:54.761412 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hgbzc" Nov 03 07:07:01.952383 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:01.952354 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-7zbzs"] Nov 03 07:07:01.955841 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:01.955823 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:02.058261 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.058224 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7pc74\"" Nov 03 07:07:02.058442 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.058414 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 07:07:02.058524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.058452 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 07:07:02.058524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.058500 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 07:07:02.059746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.059731 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-7zbzs"] Nov 03 07:07:02.065813 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.065784 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-metrics-certs\") pod \"maas-controller-579cf7c6d-7zbzs\" (UID: \"6923c64b-38c6-49b8-b3ef-108833b702c7\") " pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:02.065935 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.065907 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db7nd\" (UniqueName: \"kubernetes.io/projected/6923c64b-38c6-49b8-b3ef-108833b702c7-kube-api-access-db7nd\") pod \"maas-controller-579cf7c6d-7zbzs\" (UID: \"6923c64b-38c6-49b8-b3ef-108833b702c7\") " pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:02.065992 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.065966 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-webhook-cert\") pod \"maas-controller-579cf7c6d-7zbzs\" (UID: \"6923c64b-38c6-49b8-b3ef-108833b702c7\") " pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:02.166684 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.166657 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-webhook-cert\") pod \"maas-controller-579cf7c6d-7zbzs\" (UID: \"6923c64b-38c6-49b8-b3ef-108833b702c7\") " pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:02.166867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.166699 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-metrics-certs\") pod \"maas-controller-579cf7c6d-7zbzs\" (UID: \"6923c64b-38c6-49b8-b3ef-108833b702c7\") " pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:02.166867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.166780 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db7nd\" (UniqueName: \"kubernetes.io/projected/6923c64b-38c6-49b8-b3ef-108833b702c7-kube-api-access-db7nd\") pod \"maas-controller-579cf7c6d-7zbzs\" (UID: \"6923c64b-38c6-49b8-b3ef-108833b702c7\") " pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:02.169348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.169324 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-webhook-cert\") pod \"maas-controller-579cf7c6d-7zbzs\" (UID: \"6923c64b-38c6-49b8-b3ef-108833b702c7\") " pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:02.169465 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.169357 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-metrics-certs\") pod \"maas-controller-579cf7c6d-7zbzs\" (UID: \"6923c64b-38c6-49b8-b3ef-108833b702c7\") " pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:02.229453 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.229389 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db7nd\" (UniqueName: \"kubernetes.io/projected/6923c64b-38c6-49b8-b3ef-108833b702c7-kube-api-access-db7nd\") pod \"maas-controller-579cf7c6d-7zbzs\" (UID: \"6923c64b-38c6-49b8-b3ef-108833b702c7\") " pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:02.267440 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.267410 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:02.641071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.640991 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-7zbzs"] Nov 03 07:07:02.643728 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:07:02.643680 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6923c64b_38c6_49b8_b3ef_108833b702c7.slice/crio-8d27b48b230050e87432955583f2bccd3c529bbf0d4fdfa774daea916a00d05e WatchSource:0}: Error finding container 8d27b48b230050e87432955583f2bccd3c529bbf0d4fdfa774daea916a00d05e: Status 404 returned error can't find the container with id 8d27b48b230050e87432955583f2bccd3c529bbf0d4fdfa774daea916a00d05e Nov 03 07:07:02.788878 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:02.788828 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-7zbzs" event={"ID":"6923c64b-38c6-49b8-b3ef-108833b702c7","Type":"ContainerStarted","Data":"8d27b48b230050e87432955583f2bccd3c529bbf0d4fdfa774daea916a00d05e"} Nov 03 07:07:07.808157 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:07.808127 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 07:07:08.773337 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:07:08.773300 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:07:08.815630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:08.815589 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-7zbzs" event={"ID":"6923c64b-38c6-49b8-b3ef-108833b702c7","Type":"ContainerStarted","Data":"49d7e08cd2a5ef635a01d2512b6a4be4fb0a1a1c145330b86820a96672003491"} Nov 03 07:07:08.816084 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:08.815706 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:08.936031 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:08.935979 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-7zbzs" podStartSLOduration=2.824454973 podStartE2EDuration="7.935965195s" totalImagesPullingTime="5.111510222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:07:08.935264178 +0000 UTC m=+484.127805929" watchObservedRunningTime="2026-11-03 07:07:08.935965195 +0000 UTC m=+484.128506957" Nov 03 07:07:11.170534 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.170504 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9"] Nov 03 07:07:11.174530 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.174508 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.228767 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.228741 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 07:07:11.228897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.228793 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 07:07:11.228897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.228806 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 07:07:11.244355 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.244330 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/784a166b-28e5-4550-b5ab-0dad59624f54-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.244474 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.244367 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj5pz\" (UniqueName: \"kubernetes.io/projected/784a166b-28e5-4550-b5ab-0dad59624f54-kube-api-access-rj5pz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.244511 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.244484 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/784a166b-28e5-4550-b5ab-0dad59624f54-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.244543 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.244517 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/784a166b-28e5-4550-b5ab-0dad59624f54-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.277179 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.277146 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-9bbkp\"" Nov 03 07:07:11.278578 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.278553 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9"] Nov 03 07:07:11.345078 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.345052 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5pz\" (UniqueName: \"kubernetes.io/projected/784a166b-28e5-4550-b5ab-0dad59624f54-kube-api-access-rj5pz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.345214 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.345121 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/784a166b-28e5-4550-b5ab-0dad59624f54-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.345214 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.345154 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/784a166b-28e5-4550-b5ab-0dad59624f54-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.345336 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.345211 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/784a166b-28e5-4550-b5ab-0dad59624f54-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.345336 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:07:11.345250 4409 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 03 07:07:11.345336 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:07:11.345303 4409 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 07:07:11.345336 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:07:11.345316 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/784a166b-28e5-4550-b5ab-0dad59624f54-tls-cert podName:784a166b-28e5-4550-b5ab-0dad59624f54 nodeName:}" failed. No retries permitted until 2026-11-03 07:07:11.845299357 +0000 UTC m=+487.037841100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/784a166b-28e5-4550-b5ab-0dad59624f54-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" (UID: "784a166b-28e5-4550-b5ab-0dad59624f54") : secret "usage-logs-tenancy-proxy-tls" not found Nov 03 07:07:11.345545 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:07:11.345363 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/784a166b-28e5-4550-b5ab-0dad59624f54-proxy-source podName:784a166b-28e5-4550-b5ab-0dad59624f54 nodeName:}" failed. No retries permitted until 2026-11-03 07:07:11.845347311 +0000 UTC m=+487.037889042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/784a166b-28e5-4550-b5ab-0dad59624f54-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" (UID: "784a166b-28e5-4550-b5ab-0dad59624f54") : configmap "usage-tenancy-proxy-source" not found Nov 03 07:07:11.345943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.345921 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/784a166b-28e5-4550-b5ab-0dad59624f54-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.409391 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.409358 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj5pz\" (UniqueName: \"kubernetes.io/projected/784a166b-28e5-4550-b5ab-0dad59624f54-kube-api-access-rj5pz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.848620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.848596 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/784a166b-28e5-4550-b5ab-0dad59624f54-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.848895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.848628 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/784a166b-28e5-4550-b5ab-0dad59624f54-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.849740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.849694 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/784a166b-28e5-4550-b5ab-0dad59624f54-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:11.851199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:11.851179 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/784a166b-28e5-4550-b5ab-0dad59624f54-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7pvz9\" (UID: \"784a166b-28e5-4550-b5ab-0dad59624f54\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:12.087331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:12.087302 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:12.467082 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:12.466998 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9"] Nov 03 07:07:12.469835 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:07:12.469805 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784a166b_28e5_4550_b5ab_0dad59624f54.slice/crio-447147d82e6b3b97d9561df71ab61efb6d4adf577a946128e5eb849f68bd19d3 WatchSource:0}: Error finding container 447147d82e6b3b97d9561df71ab61efb6d4adf577a946128e5eb849f68bd19d3: Status 404 returned error can't find the container with id 447147d82e6b3b97d9561df71ab61efb6d4adf577a946128e5eb849f68bd19d3 Nov 03 07:07:12.838363 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:12.833761 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" event={"ID":"784a166b-28e5-4550-b5ab-0dad59624f54","Type":"ContainerStarted","Data":"447147d82e6b3b97d9561df71ab61efb6d4adf577a946128e5eb849f68bd19d3"} Nov 03 07:07:19.733128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:19.733030 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-7zbzs"] Nov 03 07:07:19.733699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:19.733233 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-7zbzs" podUID="6923c64b-38c6-49b8-b3ef-108833b702c7" containerName="manager" containerID="cri-o://49d7e08cd2a5ef635a01d2512b6a4be4fb0a1a1c145330b86820a96672003491" gracePeriod=30 Nov 03 07:07:19.735599 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:19.735575 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:19.875743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:19.875051 4409 generic.go:356] "Generic (PLEG): container finished" podID="6923c64b-38c6-49b8-b3ef-108833b702c7" containerID="49d7e08cd2a5ef635a01d2512b6a4be4fb0a1a1c145330b86820a96672003491" exitCode=0 Nov 03 07:07:19.875743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:19.875155 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-7zbzs" event={"ID":"6923c64b-38c6-49b8-b3ef-108833b702c7","Type":"ContainerDied","Data":"49d7e08cd2a5ef635a01d2512b6a4be4fb0a1a1c145330b86820a96672003491"} Nov 03 07:07:20.047784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.047703 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:20.115399 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.115371 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6923c64b-38c6-49b8-b3ef-108833b702c7-kube-api-access-db7nd\" (UniqueName: \"kubernetes.io/projected/6923c64b-38c6-49b8-b3ef-108833b702c7-kube-api-access-db7nd\") pod \"6923c64b-38c6-49b8-b3ef-108833b702c7\" (UID: \"6923c64b-38c6-49b8-b3ef-108833b702c7\") " Nov 03 07:07:20.115571 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.115430 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-webhook-cert\") pod \"6923c64b-38c6-49b8-b3ef-108833b702c7\" (UID: \"6923c64b-38c6-49b8-b3ef-108833b702c7\") " Nov 03 07:07:20.115571 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.115483 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-metrics-certs\") pod \"6923c64b-38c6-49b8-b3ef-108833b702c7\" (UID: \"6923c64b-38c6-49b8-b3ef-108833b702c7\") " Nov 03 07:07:20.119812 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.119776 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6923c64b-38c6-49b8-b3ef-108833b702c7-kube-api-access-db7nd" pod "6923c64b-38c6-49b8-b3ef-108833b702c7" (UID: "6923c64b-38c6-49b8-b3ef-108833b702c7"). InnerVolumeSpecName "kube-api-access-db7nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:07:20.119998 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.119969 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-webhook-cert" pod "6923c64b-38c6-49b8-b3ef-108833b702c7" (UID: "6923c64b-38c6-49b8-b3ef-108833b702c7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:07:20.120117 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.120027 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-metrics-certs" pod "6923c64b-38c6-49b8-b3ef-108833b702c7" (UID: "6923c64b-38c6-49b8-b3ef-108833b702c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:07:20.217029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.216931 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db7nd\" (UniqueName: \"kubernetes.io/projected/6923c64b-38c6-49b8-b3ef-108833b702c7-kube-api-access-db7nd\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:07:20.217194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.217034 4409 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-webhook-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:07:20.217194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.217052 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6923c64b-38c6-49b8-b3ef-108833b702c7-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:07:20.881669 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.881628 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-7zbzs" event={"ID":"6923c64b-38c6-49b8-b3ef-108833b702c7","Type":"ContainerDied","Data":"8d27b48b230050e87432955583f2bccd3c529bbf0d4fdfa774daea916a00d05e"} Nov 03 07:07:20.881669 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.881656 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-7zbzs" Nov 03 07:07:20.882218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:20.881698 4409 scope.go:122] "RemoveContainer" containerID="49d7e08cd2a5ef635a01d2512b6a4be4fb0a1a1c145330b86820a96672003491" Nov 03 07:07:21.052568 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:21.052526 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-7zbzs"] Nov 03 07:07:21.101490 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:21.101461 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-7zbzs"] Nov 03 07:07:22.911038 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:22.911005 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6923c64b-38c6-49b8-b3ef-108833b702c7" path="/var/lib/kubelet/pods/6923c64b-38c6-49b8-b3ef-108833b702c7/volumes" Nov 03 07:07:24.807641 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.807603 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx"] Nov 03 07:07:24.808102 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.808002 4409 cpu_manager.go:414] "containerMap: removing container" podUID="6923c64b-38c6-49b8-b3ef-108833b702c7" containerName="manager" Nov 03 07:07:24.808102 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.808022 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6923c64b-38c6-49b8-b3ef-108833b702c7" containerName="manager" Nov 03 07:07:24.808102 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.808094 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="6923c64b-38c6-49b8-b3ef-108833b702c7" containerName="manager" Nov 03 07:07:24.828382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.828360 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.859070 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.859048 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7410ef1-cc43-4ca1-b173-546497e70c1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.859193 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.859096 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7410ef1-cc43-4ca1-b173-546497e70c1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.859193 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.859117 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7410ef1-cc43-4ca1-b173-546497e70c1a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.859302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.859195 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7410ef1-cc43-4ca1-b173-546497e70c1a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.859302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.859242 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqqw5\" (UniqueName: \"kubernetes.io/projected/f7410ef1-cc43-4ca1-b173-546497e70c1a-kube-api-access-gqqw5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.859423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.859349 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7410ef1-cc43-4ca1-b173-546497e70c1a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.859423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.859371 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7410ef1-cc43-4ca1-b173-546497e70c1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.919870 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.919800 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx"] Nov 03 07:07:24.960420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.960389 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7410ef1-cc43-4ca1-b173-546497e70c1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.960593 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.960441 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7410ef1-cc43-4ca1-b173-546497e70c1a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.960593 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.960476 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7410ef1-cc43-4ca1-b173-546497e70c1a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.960593 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.960503 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqqw5\" (UniqueName: \"kubernetes.io/projected/f7410ef1-cc43-4ca1-b173-546497e70c1a-kube-api-access-gqqw5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.960593 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.960563 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7410ef1-cc43-4ca1-b173-546497e70c1a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.960849 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.960594 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7410ef1-cc43-4ca1-b173-546497e70c1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.960849 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.960660 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7410ef1-cc43-4ca1-b173-546497e70c1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.961459 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.961432 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7410ef1-cc43-4ca1-b173-546497e70c1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.963901 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.963412 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7410ef1-cc43-4ca1-b173-546497e70c1a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.963901 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.963850 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7410ef1-cc43-4ca1-b173-546497e70c1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.964062 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.964046 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7410ef1-cc43-4ca1-b173-546497e70c1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:24.964967 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:24.964938 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7410ef1-cc43-4ca1-b173-546497e70c1a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:25.023579 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:25.023554 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7410ef1-cc43-4ca1-b173-546497e70c1a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:25.023811 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:25.023791 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqqw5\" (UniqueName: \"kubernetes.io/projected/f7410ef1-cc43-4ca1-b173-546497e70c1a-kube-api-access-gqqw5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c8rtx\" (UID: \"f7410ef1-cc43-4ca1-b173-546497e70c1a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:25.142110 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:25.142037 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:25.575558 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:25.575534 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx"] Nov 03 07:07:25.577446 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:07:25.577422 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7410ef1_cc43_4ca1_b173_546497e70c1a.slice/crio-23176c909511ff1f609f2e0bb5b83b3a650cda3f6f23df1b419e6d53a3573ad3 WatchSource:0}: Error finding container 23176c909511ff1f609f2e0bb5b83b3a650cda3f6f23df1b419e6d53a3573ad3: Status 404 returned error can't find the container with id 23176c909511ff1f609f2e0bb5b83b3a650cda3f6f23df1b419e6d53a3573ad3 Nov 03 07:07:25.579770 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:25.579741 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:07:25.579874 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:25.579794 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:07:25.905302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:25.905210 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" event={"ID":"f7410ef1-cc43-4ca1-b173-546497e70c1a","Type":"ContainerStarted","Data":"50a68f638261b68d9eda0746009ec598a1ba7b34e40919c31265c8a3773b5523"} Nov 03 07:07:25.905302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:25.905265 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" event={"ID":"f7410ef1-cc43-4ca1-b173-546497e70c1a","Type":"ContainerStarted","Data":"23176c909511ff1f609f2e0bb5b83b3a650cda3f6f23df1b419e6d53a3573ad3"} Nov 03 07:07:25.907214 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:25.907187 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" event={"ID":"784a166b-28e5-4550-b5ab-0dad59624f54","Type":"ContainerStarted","Data":"4e50478fe142294b919aa7c8d8642d432d9090b6224f0974db070ccff542ba2c"} Nov 03 07:07:25.907354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:25.907231 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:26.029877 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:26.029832 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" podStartSLOduration=2.029817646 podStartE2EDuration="2.029817646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:07:26.027926411 +0000 UTC m=+501.220468161" watchObservedRunningTime="2026-11-03 07:07:26.029817646 +0000 UTC m=+501.222359393" Nov 03 07:07:26.155416 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:26.155313 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" podStartSLOduration=2.30962824 podStartE2EDuration="15.155298848s" totalImagesPullingTime="12.845670608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:07:26.14751332 +0000 UTC m=+501.340055073" watchObservedRunningTime="2026-11-03 07:07:26.155298848 +0000 UTC m=+501.347840598" Nov 03 07:07:26.911229 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:26.911199 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:26.912883 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:26.912859 4409 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-c8rtx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:07:26.912996 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:26.912915 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" podUID="f7410ef1-cc43-4ca1-b173-546497e70c1a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:07:27.915939 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:27.915909 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c8rtx" Nov 03 07:07:28.138640 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.138608 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f"] Nov 03 07:07:28.138965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.138917 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" podUID="0e08d1a8-a40d-481c-b128-d1987d3d6cd1" containerName="discovery" containerID="cri-o://dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029" gracePeriod=30 Nov 03 07:07:28.486838 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.486817 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:07:28.593938 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.593909 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-ca-configmap\") pod \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " Nov 03 07:07:28.593938 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.593939 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-kubeconfig\") pod \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " Nov 03 07:07:28.594194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.593968 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-dns-cert\") pod \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " Nov 03 07:07:28.594194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.594007 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-local-certs\") pod \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " Nov 03 07:07:28.594194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.594037 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-cacerts\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-cacerts\") pod \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " Nov 03 07:07:28.594194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.594059 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-kube-api-access-dh7t8\" (UniqueName: \"kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-kube-api-access-dh7t8\") pod \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " Nov 03 07:07:28.594395 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.594212 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-token\" (UniqueName: \"kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-token\") pod \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\" (UID: \"0e08d1a8-a40d-481c-b128-d1987d3d6cd1\") " Nov 03 07:07:28.594395 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.594343 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-ca-configmap" pod "0e08d1a8-a40d-481c-b128-d1987d3d6cd1" (UID: "0e08d1a8-a40d-481c-b128-d1987d3d6cd1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:07:28.594527 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.594503 4409 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-ca-configmap\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:07:28.596672 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.596632 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-cacerts" pod "0e08d1a8-a40d-481c-b128-d1987d3d6cd1" (UID: "0e08d1a8-a40d-481c-b128-d1987d3d6cd1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:07:28.596827 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.596741 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-token" pod "0e08d1a8-a40d-481c-b128-d1987d3d6cd1" (UID: "0e08d1a8-a40d-481c-b128-d1987d3d6cd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:07:28.596871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.596831 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-local-certs" pod "0e08d1a8-a40d-481c-b128-d1987d3d6cd1" (UID: "0e08d1a8-a40d-481c-b128-d1987d3d6cd1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:07:28.596871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.596832 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-kubeconfig" pod "0e08d1a8-a40d-481c-b128-d1987d3d6cd1" (UID: "0e08d1a8-a40d-481c-b128-d1987d3d6cd1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:07:28.596995 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.596974 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-dns-cert" pod "0e08d1a8-a40d-481c-b128-d1987d3d6cd1" (UID: "0e08d1a8-a40d-481c-b128-d1987d3d6cd1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:07:28.596995 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.596982 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-kube-api-access-dh7t8" pod "0e08d1a8-a40d-481c-b128-d1987d3d6cd1" (UID: "0e08d1a8-a40d-481c-b128-d1987d3d6cd1"). InnerVolumeSpecName "kube-api-access-dh7t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:07:28.695301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.695272 4409 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-kubeconfig\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:07:28.695301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.695298 4409 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-csr-dns-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:07:28.695301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.695307 4409 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-local-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:07:28.695301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.695318 4409 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-cacerts\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:07:28.695570 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.695327 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh7t8\" (UniqueName: \"kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-kube-api-access-dh7t8\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:07:28.695570 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.695336 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e08d1a8-a40d-481c-b128-d1987d3d6cd1-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:07:28.921097 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.921068 4409 generic.go:356] "Generic (PLEG): container finished" podID="0e08d1a8-a40d-481c-b128-d1987d3d6cd1" containerID="dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029" exitCode=0 Nov 03 07:07:28.921448 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.921121 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" Nov 03 07:07:28.921448 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.921138 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" event={"ID":"0e08d1a8-a40d-481c-b128-d1987d3d6cd1","Type":"ContainerDied","Data":"dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029"} Nov 03 07:07:28.921448 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.921191 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f" event={"ID":"0e08d1a8-a40d-481c-b128-d1987d3d6cd1","Type":"ContainerDied","Data":"457b133564a72f1d27c25151eab320886c9fac1fda17bdec5091281e28a869a3"} Nov 03 07:07:28.921448 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.921217 4409 scope.go:122] "RemoveContainer" containerID="dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029" Nov 03 07:07:28.929852 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.929837 4409 scope.go:122] "RemoveContainer" containerID="dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029" Nov 03 07:07:28.930128 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:07:28.930108 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029\": container with ID starting with dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029 not found: ID does not exist" containerID="dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029" Nov 03 07:07:28.930240 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:28.930133 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029"} err="failed to get container status \"dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029\": rpc error: code = NotFound desc = could not find container \"dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029\": container with ID starting with dfb41f590a1ff46136d2db686d4fc0d38c44aba55673014d0fa6721daddfd029 not found: ID does not exist" Nov 03 07:07:29.089507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:29.089478 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f"] Nov 03 07:07:29.100407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:29.100381 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gvv6f"] Nov 03 07:07:30.911953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:30.911907 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e08d1a8-a40d-481c-b128-d1987d3d6cd1" path="/var/lib/kubelet/pods/0e08d1a8-a40d-481c-b128-d1987d3d6cd1/volumes" Nov 03 07:07:31.154919 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.154891 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lb7g2"] Nov 03 07:07:31.155283 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.155267 4409 cpu_manager.go:414] "containerMap: removing container" podUID="0e08d1a8-a40d-481c-b128-d1987d3d6cd1" containerName="discovery" Nov 03 07:07:31.155369 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.155286 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e08d1a8-a40d-481c-b128-d1987d3d6cd1" containerName="discovery" Nov 03 07:07:31.155419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.155375 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e08d1a8-a40d-481c-b128-d1987d3d6cd1" containerName="discovery" Nov 03 07:07:31.159630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.159609 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" Nov 03 07:07:31.215649 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.215622 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb50aec-86da-4891-a86f-5217b41dbd40-tls-certs\") pod \"model-serving-api-75bd89fdfc-lb7g2\" (UID: \"5bb50aec-86da-4891-a86f-5217b41dbd40\") " pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" Nov 03 07:07:31.215806 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.215788 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldg8g\" (UniqueName: \"kubernetes.io/projected/5bb50aec-86da-4891-a86f-5217b41dbd40-kube-api-access-ldg8g\") pod \"model-serving-api-75bd89fdfc-lb7g2\" (UID: \"5bb50aec-86da-4891-a86f-5217b41dbd40\") " pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" Nov 03 07:07:31.218075 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.218054 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lb7g2"] Nov 03 07:07:31.265171 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.265139 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 07:07:31.265171 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.265164 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-swwrp\"" Nov 03 07:07:31.271694 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.271671 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-ttjfs"] Nov 03 07:07:31.274850 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.274833 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:31.316858 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.316827 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbec41ca-ada9-4e0b-9753-25988976e073-cert\") pod \"odh-model-controller-769875db56-ttjfs\" (UID: \"cbec41ca-ada9-4e0b-9753-25988976e073\") " pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:31.316982 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.316871 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkqp\" (UniqueName: \"kubernetes.io/projected/cbec41ca-ada9-4e0b-9753-25988976e073-kube-api-access-6bkqp\") pod \"odh-model-controller-769875db56-ttjfs\" (UID: \"cbec41ca-ada9-4e0b-9753-25988976e073\") " pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:31.316982 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.316957 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb50aec-86da-4891-a86f-5217b41dbd40-tls-certs\") pod \"model-serving-api-75bd89fdfc-lb7g2\" (UID: \"5bb50aec-86da-4891-a86f-5217b41dbd40\") " pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" Nov 03 07:07:31.317081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.317033 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldg8g\" (UniqueName: \"kubernetes.io/projected/5bb50aec-86da-4891-a86f-5217b41dbd40-kube-api-access-ldg8g\") pod \"model-serving-api-75bd89fdfc-lb7g2\" (UID: \"5bb50aec-86da-4891-a86f-5217b41dbd40\") " pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" Nov 03 07:07:31.317081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.317053 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbec41ca-ada9-4e0b-9753-25988976e073-metrics-certs\") pod \"odh-model-controller-769875db56-ttjfs\" (UID: \"cbec41ca-ada9-4e0b-9753-25988976e073\") " pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:31.319502 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.319482 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb50aec-86da-4891-a86f-5217b41dbd40-tls-certs\") pod \"model-serving-api-75bd89fdfc-lb7g2\" (UID: \"5bb50aec-86da-4891-a86f-5217b41dbd40\") " pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" Nov 03 07:07:31.329671 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.329651 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-r84k2\"" Nov 03 07:07:31.329808 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.329763 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 07:07:31.329808 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.329763 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 07:07:31.334992 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.334964 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ttjfs"] Nov 03 07:07:31.380409 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.380382 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldg8g\" (UniqueName: \"kubernetes.io/projected/5bb50aec-86da-4891-a86f-5217b41dbd40-kube-api-access-ldg8g\") pod \"model-serving-api-75bd89fdfc-lb7g2\" (UID: \"5bb50aec-86da-4891-a86f-5217b41dbd40\") " pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" Nov 03 07:07:31.417809 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.417748 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbec41ca-ada9-4e0b-9753-25988976e073-cert\") pod \"odh-model-controller-769875db56-ttjfs\" (UID: \"cbec41ca-ada9-4e0b-9753-25988976e073\") " pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:31.417809 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.417792 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkqp\" (UniqueName: \"kubernetes.io/projected/cbec41ca-ada9-4e0b-9753-25988976e073-kube-api-access-6bkqp\") pod \"odh-model-controller-769875db56-ttjfs\" (UID: \"cbec41ca-ada9-4e0b-9753-25988976e073\") " pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:31.417980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.417850 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbec41ca-ada9-4e0b-9753-25988976e073-metrics-certs\") pod \"odh-model-controller-769875db56-ttjfs\" (UID: \"cbec41ca-ada9-4e0b-9753-25988976e073\") " pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:31.420188 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.420168 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbec41ca-ada9-4e0b-9753-25988976e073-metrics-certs\") pod \"odh-model-controller-769875db56-ttjfs\" (UID: \"cbec41ca-ada9-4e0b-9753-25988976e073\") " pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:31.420270 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.420235 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbec41ca-ada9-4e0b-9753-25988976e073-cert\") pod \"odh-model-controller-769875db56-ttjfs\" (UID: \"cbec41ca-ada9-4e0b-9753-25988976e073\") " pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:31.472070 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.472051 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" Nov 03 07:07:31.481244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.481218 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkqp\" (UniqueName: \"kubernetes.io/projected/cbec41ca-ada9-4e0b-9753-25988976e073-kube-api-access-6bkqp\") pod \"odh-model-controller-769875db56-ttjfs\" (UID: \"cbec41ca-ada9-4e0b-9753-25988976e073\") " pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:31.587866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.587840 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:31.843449 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.843426 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lb7g2"] Nov 03 07:07:31.845999 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:07:31.845975 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb50aec_86da_4891_a86f_5217b41dbd40.slice/crio-390c37e49c7c8a22bbebd335ffafb9f3bccbf52628f09ccbe1de1c5ad350a607 WatchSource:0}: Error finding container 390c37e49c7c8a22bbebd335ffafb9f3bccbf52628f09ccbe1de1c5ad350a607: Status 404 returned error can't find the container with id 390c37e49c7c8a22bbebd335ffafb9f3bccbf52628f09ccbe1de1c5ad350a607 Nov 03 07:07:31.937827 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.937803 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" event={"ID":"5bb50aec-86da-4891-a86f-5217b41dbd40","Type":"ContainerStarted","Data":"390c37e49c7c8a22bbebd335ffafb9f3bccbf52628f09ccbe1de1c5ad350a607"} Nov 03 07:07:31.961275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:31.961259 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ttjfs"] Nov 03 07:07:31.963751 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:07:31.963707 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbec41ca_ada9_4e0b_9753_25988976e073.slice/crio-c2afadce542a461325cb85c1a2e0e05012d6cc08a80a9f920e6c4d6e021b1558 WatchSource:0}: Error finding container c2afadce542a461325cb85c1a2e0e05012d6cc08a80a9f920e6c4d6e021b1558: Status 404 returned error can't find the container with id c2afadce542a461325cb85c1a2e0e05012d6cc08a80a9f920e6c4d6e021b1558 Nov 03 07:07:32.943449 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:32.943410 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ttjfs" event={"ID":"cbec41ca-ada9-4e0b-9753-25988976e073","Type":"ContainerStarted","Data":"c2afadce542a461325cb85c1a2e0e05012d6cc08a80a9f920e6c4d6e021b1558"} Nov 03 07:07:34.955691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:34.955649 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" event={"ID":"5bb50aec-86da-4891-a86f-5217b41dbd40","Type":"ContainerStarted","Data":"5f3fd114b86d011cfe5a11bf390d6aecf5485469dc3cff2df63ed15691510509"} Nov 03 07:07:34.956197 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:34.955773 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" Nov 03 07:07:35.075773 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:35.075701 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" podStartSLOduration=1.499994676 podStartE2EDuration="4.075686047s" totalImagesPullingTime="2.575691371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:07:35.074397099 +0000 UTC m=+510.266938851" watchObservedRunningTime="2026-11-03 07:07:35.075686047 +0000 UTC m=+510.268227797" Nov 03 07:07:35.961780 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:35.961751 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ttjfs" event={"ID":"cbec41ca-ada9-4e0b-9753-25988976e073","Type":"ContainerStarted","Data":"f2112483284e2b6b5346726dfe284cc18ac180c55e994d37947819dfe8f27e5c"} Nov 03 07:07:35.962211 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:35.961870 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:36.081282 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:36.081225 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-ttjfs" podStartSLOduration=1.74991344 podStartE2EDuration="5.081211031s" totalImagesPullingTime="3.331297591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:07:36.07976771 +0000 UTC m=+511.272309459" watchObservedRunningTime="2026-11-03 07:07:36.081211031 +0000 UTC m=+511.273752780" Nov 03 07:07:36.916836 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:36.916810 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7pvz9" Nov 03 07:07:38.774266 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:07:38.774230 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:07:44.749153 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.749122 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8"] Nov 03 07:07:44.753749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.753706 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:44.838292 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.838264 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-metrics-certs\") pod \"maas-api-6d669c7fcd-mdhv8\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:44.838508 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.838304 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdlff\" (UniqueName: \"kubernetes.io/projected/a34864e6-d055-43c3-b1fe-d17f098124f9-kube-api-access-jdlff\") pod \"maas-api-6d669c7fcd-mdhv8\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:44.838508 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.838482 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-maas-api-tls\") pod \"maas-api-6d669c7fcd-mdhv8\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:44.838648 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.838526 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34864e6-d055-43c3-b1fe-d17f098124f9-openshift-service-ca\") pod \"maas-api-6d669c7fcd-mdhv8\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:44.856735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.856686 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 07:07:44.856910 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.856779 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 07:07:44.856910 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.856833 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6zz26\"" Nov 03 07:07:44.861464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.861447 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8"] Nov 03 07:07:44.939441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.939402 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-maas-api-tls\") pod \"maas-api-6d669c7fcd-mdhv8\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:44.939441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.939443 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34864e6-d055-43c3-b1fe-d17f098124f9-openshift-service-ca\") pod \"maas-api-6d669c7fcd-mdhv8\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:44.939687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.939580 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-metrics-certs\") pod \"maas-api-6d669c7fcd-mdhv8\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:44.939687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.939634 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdlff\" (UniqueName: \"kubernetes.io/projected/a34864e6-d055-43c3-b1fe-d17f098124f9-kube-api-access-jdlff\") pod \"maas-api-6d669c7fcd-mdhv8\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:44.940032 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.940015 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34864e6-d055-43c3-b1fe-d17f098124f9-openshift-service-ca\") pod \"maas-api-6d669c7fcd-mdhv8\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:44.942030 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.942008 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-metrics-certs\") pod \"maas-api-6d669c7fcd-mdhv8\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:44.942132 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:44.942062 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-maas-api-tls\") pod \"maas-api-6d669c7fcd-mdhv8\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:45.008286 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:45.008204 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdlff\" (UniqueName: \"kubernetes.io/projected/a34864e6-d055-43c3-b1fe-d17f098124f9-kube-api-access-jdlff\") pod \"maas-api-6d669c7fcd-mdhv8\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:45.065673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:45.065635 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:45.442165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:45.442139 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8"] Nov 03 07:07:45.444575 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:07:45.444545 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34864e6_d055_43c3_b1fe_d17f098124f9.slice/crio-a93a76c325b08fb67fc738eeab68709858d61f609fc1161aee262bdc7cb041e7 WatchSource:0}: Error finding container a93a76c325b08fb67fc738eeab68709858d61f609fc1161aee262bdc7cb041e7: Status 404 returned error can't find the container with id a93a76c325b08fb67fc738eeab68709858d61f609fc1161aee262bdc7cb041e7 Nov 03 07:07:45.966845 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:45.966822 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-lb7g2" Nov 03 07:07:46.006408 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:46.006365 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" event={"ID":"a34864e6-d055-43c3-b1fe-d17f098124f9","Type":"ContainerStarted","Data":"a93a76c325b08fb67fc738eeab68709858d61f609fc1161aee262bdc7cb041e7"} Nov 03 07:07:46.968123 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:46.968091 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-ttjfs" Nov 03 07:07:51.030828 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:51.030789 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" event={"ID":"a34864e6-d055-43c3-b1fe-d17f098124f9","Type":"ContainerStarted","Data":"2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34"} Nov 03 07:07:51.031319 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:51.030935 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:51.151788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:51.151703 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" podStartSLOduration=2.617650593 podStartE2EDuration="7.151684343s" totalImagesPullingTime="4.53403375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:07:51.150943818 +0000 UTC m=+526.343485568" watchObservedRunningTime="2026-11-03 07:07:51.151684343 +0000 UTC m=+526.344226094" Nov 03 07:07:57.040378 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:57.040349 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:07:58.095002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.094970 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh"] Nov 03 07:07:58.197814 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.197786 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh"] Nov 03 07:07:58.197968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.197880 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.349905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.349826 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9255898a-7131-44de-a617-f2c92d75714b-openshift-service-ca\") pod \"maas-api-78f9b5fd67-8vnsh\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.349905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.349896 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrbvz\" (UniqueName: \"kubernetes.io/projected/9255898a-7131-44de-a617-f2c92d75714b-kube-api-access-xrbvz\") pod \"maas-api-78f9b5fd67-8vnsh\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.350091 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.350003 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-maas-api-tls\") pod \"maas-api-78f9b5fd67-8vnsh\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.350091 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.350042 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-metrics-certs\") pod \"maas-api-78f9b5fd67-8vnsh\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.450852 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.450823 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrbvz\" (UniqueName: \"kubernetes.io/projected/9255898a-7131-44de-a617-f2c92d75714b-kube-api-access-xrbvz\") pod \"maas-api-78f9b5fd67-8vnsh\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.450980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.450859 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-maas-api-tls\") pod \"maas-api-78f9b5fd67-8vnsh\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.450980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.450879 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-metrics-certs\") pod \"maas-api-78f9b5fd67-8vnsh\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.450980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.450904 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9255898a-7131-44de-a617-f2c92d75714b-openshift-service-ca\") pod \"maas-api-78f9b5fd67-8vnsh\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.451612 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.451513 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9255898a-7131-44de-a617-f2c92d75714b-openshift-service-ca\") pod \"maas-api-78f9b5fd67-8vnsh\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.453274 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.453253 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-metrics-certs\") pod \"maas-api-78f9b5fd67-8vnsh\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.453389 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.453373 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-maas-api-tls\") pod \"maas-api-78f9b5fd67-8vnsh\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.513707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.513680 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrbvz\" (UniqueName: \"kubernetes.io/projected/9255898a-7131-44de-a617-f2c92d75714b-kube-api-access-xrbvz\") pod \"maas-api-78f9b5fd67-8vnsh\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.517535 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.517513 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:07:58.902947 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:58.902861 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh"] Nov 03 07:07:58.905570 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:07:58.905537 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9255898a_7131_44de_a617_f2c92d75714b.slice/crio-03f63a38ca64f8dc7e8081e77f56fa0ab192d2b6356750a40e97b366827f0466 WatchSource:0}: Error finding container 03f63a38ca64f8dc7e8081e77f56fa0ab192d2b6356750a40e97b366827f0466: Status 404 returned error can't find the container with id 03f63a38ca64f8dc7e8081e77f56fa0ab192d2b6356750a40e97b366827f0466 Nov 03 07:07:59.065581 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:07:59.065548 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" event={"ID":"9255898a-7131-44de-a617-f2c92d75714b","Type":"ContainerStarted","Data":"03f63a38ca64f8dc7e8081e77f56fa0ab192d2b6356750a40e97b366827f0466"} Nov 03 07:08:00.073615 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:00.073573 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" event={"ID":"9255898a-7131-44de-a617-f2c92d75714b","Type":"ContainerStarted","Data":"acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b"} Nov 03 07:08:00.074101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:00.073748 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:08:00.217305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:00.217217 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" podStartSLOduration=1.20875225 podStartE2EDuration="2.217201949s" totalImagesPullingTime="1.008449699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:08:00.215835397 +0000 UTC m=+535.408377150" watchObservedRunningTime="2026-11-03 07:08:00.217201949 +0000 UTC m=+535.409743699" Nov 03 07:08:02.381365 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:02.381334 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mnd8x"] Nov 03 07:08:02.381882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:02.381523 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" podUID="fabbdc26-a95a-401a-b787-023b23fd24e7" containerName="authorino" containerID="cri-o://1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165" gracePeriod=30 Nov 03 07:08:02.620939 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:02.620918 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" Nov 03 07:08:02.789208 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:02.789139 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fabbdc26-a95a-401a-b787-023b23fd24e7-kube-api-access-hc722\" (UniqueName: \"kubernetes.io/projected/fabbdc26-a95a-401a-b787-023b23fd24e7-kube-api-access-hc722\") pod \"fabbdc26-a95a-401a-b787-023b23fd24e7\" (UID: \"fabbdc26-a95a-401a-b787-023b23fd24e7\") " Nov 03 07:08:02.791215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:02.791188 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabbdc26-a95a-401a-b787-023b23fd24e7-kube-api-access-hc722" pod "fabbdc26-a95a-401a-b787-023b23fd24e7" (UID: "fabbdc26-a95a-401a-b787-023b23fd24e7"). InnerVolumeSpecName "kube-api-access-hc722". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:08:02.890524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:02.890501 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc722\" (UniqueName: \"kubernetes.io/projected/fabbdc26-a95a-401a-b787-023b23fd24e7-kube-api-access-hc722\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:08:03.088948 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:03.088920 4409 generic.go:356] "Generic (PLEG): container finished" podID="fabbdc26-a95a-401a-b787-023b23fd24e7" containerID="1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165" exitCode=0 Nov 03 07:08:03.089100 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:03.088985 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" Nov 03 07:08:03.089100 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:03.089005 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" event={"ID":"fabbdc26-a95a-401a-b787-023b23fd24e7","Type":"ContainerDied","Data":"1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165"} Nov 03 07:08:03.089100 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:03.089057 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mnd8x" event={"ID":"fabbdc26-a95a-401a-b787-023b23fd24e7","Type":"ContainerDied","Data":"c87c093bedc42befbfbb4782793fce20452eed129bdc019eb2eae452b32babfa"} Nov 03 07:08:03.089100 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:03.089081 4409 scope.go:122] "RemoveContainer" containerID="1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165" Nov 03 07:08:03.098427 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:03.098406 4409 scope.go:122] "RemoveContainer" containerID="1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165" Nov 03 07:08:03.098722 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:08:03.098690 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165\": container with ID starting with 1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165 not found: ID does not exist" containerID="1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165" Nov 03 07:08:03.098825 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:03.098806 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165"} err="failed to get container status \"1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165\": rpc error: code = NotFound desc = could not find container \"1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165\": container with ID starting with 1bda071b271a76bc85842da79b6dcf8ae33b5e9227f88512320c998c71116165 not found: ID does not exist" Nov 03 07:08:03.256346 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:03.256322 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mnd8x"] Nov 03 07:08:03.259194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:03.259165 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mnd8x"] Nov 03 07:08:04.911227 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:04.911187 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fabbdc26-a95a-401a-b787-023b23fd24e7" path="/var/lib/kubelet/pods/fabbdc26-a95a-401a-b787-023b23fd24e7/volumes" Nov 03 07:08:06.083559 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.083530 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 07:08:06.255777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.255747 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8"] Nov 03 07:08:06.255992 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.255960 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" podUID="a34864e6-d055-43c3-b1fe-d17f098124f9" containerName="maas-api" containerID="cri-o://2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34" gracePeriod=30 Nov 03 07:08:06.517211 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.517182 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:08:06.617295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.617271 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-metrics-certs\") pod \"a34864e6-d055-43c3-b1fe-d17f098124f9\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " Nov 03 07:08:06.617431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.617302 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a34864e6-d055-43c3-b1fe-d17f098124f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34864e6-d055-43c3-b1fe-d17f098124f9-openshift-service-ca\") pod \"a34864e6-d055-43c3-b1fe-d17f098124f9\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " Nov 03 07:08:06.617431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.617353 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34864e6-d055-43c3-b1fe-d17f098124f9-kube-api-access-jdlff\" (UniqueName: \"kubernetes.io/projected/a34864e6-d055-43c3-b1fe-d17f098124f9-kube-api-access-jdlff\") pod \"a34864e6-d055-43c3-b1fe-d17f098124f9\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " Nov 03 07:08:06.617431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.617387 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-maas-api-tls\") pod \"a34864e6-d055-43c3-b1fe-d17f098124f9\" (UID: \"a34864e6-d055-43c3-b1fe-d17f098124f9\") " Nov 03 07:08:06.617776 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.617752 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34864e6-d055-43c3-b1fe-d17f098124f9-openshift-service-ca" pod "a34864e6-d055-43c3-b1fe-d17f098124f9" (UID: "a34864e6-d055-43c3-b1fe-d17f098124f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:08:06.619637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.619615 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-metrics-certs" pod "a34864e6-d055-43c3-b1fe-d17f098124f9" (UID: "a34864e6-d055-43c3-b1fe-d17f098124f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:08:06.619704 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.619676 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-maas-api-tls" pod "a34864e6-d055-43c3-b1fe-d17f098124f9" (UID: "a34864e6-d055-43c3-b1fe-d17f098124f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:08:06.619776 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.619758 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34864e6-d055-43c3-b1fe-d17f098124f9-kube-api-access-jdlff" pod "a34864e6-d055-43c3-b1fe-d17f098124f9" (UID: "a34864e6-d055-43c3-b1fe-d17f098124f9"). InnerVolumeSpecName "kube-api-access-jdlff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:08:06.718450 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.718388 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:08:06.718450 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.718412 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34864e6-d055-43c3-b1fe-d17f098124f9-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:08:06.718450 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.718421 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34864e6-d055-43c3-b1fe-d17f098124f9-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:08:06.718450 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:06.718429 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdlff\" (UniqueName: \"kubernetes.io/projected/a34864e6-d055-43c3-b1fe-d17f098124f9-kube-api-access-jdlff\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:08:07.106971 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:07.106944 4409 generic.go:356] "Generic (PLEG): container finished" podID="a34864e6-d055-43c3-b1fe-d17f098124f9" containerID="2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34" exitCode=0 Nov 03 07:08:07.107349 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:07.107002 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" Nov 03 07:08:07.107349 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:07.107036 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" event={"ID":"a34864e6-d055-43c3-b1fe-d17f098124f9","Type":"ContainerDied","Data":"2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34"} Nov 03 07:08:07.107349 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:07.107101 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8" event={"ID":"a34864e6-d055-43c3-b1fe-d17f098124f9","Type":"ContainerDied","Data":"a93a76c325b08fb67fc738eeab68709858d61f609fc1161aee262bdc7cb041e7"} Nov 03 07:08:07.107349 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:07.107123 4409 scope.go:122] "RemoveContainer" containerID="2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34" Nov 03 07:08:07.115584 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:07.115566 4409 scope.go:122] "RemoveContainer" containerID="2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34" Nov 03 07:08:07.115877 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:08:07.115853 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34\": container with ID starting with 2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34 not found: ID does not exist" containerID="2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34" Nov 03 07:08:07.115951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:07.115890 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34"} err="failed to get container status \"2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34\": rpc error: code = NotFound desc = could not find container \"2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34\": container with ID starting with 2d2b54cb30a42d42d3f6a3afe4083ec749ce2af284470d5d349af1f1f7ad2c34 not found: ID does not exist" Nov 03 07:08:07.274703 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:07.274674 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8"] Nov 03 07:08:07.323969 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:07.323936 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-mdhv8"] Nov 03 07:08:08.772375 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:08:08.772342 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:08:08.910258 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:08:08.910222 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a34864e6-d055-43c3-b1fe-d17f098124f9" path="/var/lib/kubelet/pods/a34864e6-d055-43c3-b1fe-d17f098124f9/volumes" Nov 03 07:08:38.773277 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:08:38.773170 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:09:04.944310 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:04.944283 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:09:04.946707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:04.946685 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:09:08.772821 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:09:08.772785 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:09:38.773881 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:09:38.773838 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:09:40.031908 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.031877 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-sckng"] Nov 03 07:09:40.032346 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.032230 4409 cpu_manager.go:414] "containerMap: removing container" podUID="a34864e6-d055-43c3-b1fe-d17f098124f9" containerName="maas-api" Nov 03 07:09:40.032346 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.032242 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a34864e6-d055-43c3-b1fe-d17f098124f9" containerName="maas-api" Nov 03 07:09:40.032346 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.032254 4409 cpu_manager.go:414] "containerMap: removing container" podUID="fabbdc26-a95a-401a-b787-023b23fd24e7" containerName="authorino" Nov 03 07:09:40.032346 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.032261 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabbdc26-a95a-401a-b787-023b23fd24e7" containerName="authorino" Nov 03 07:09:40.032346 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.032324 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="a34864e6-d055-43c3-b1fe-d17f098124f9" containerName="maas-api" Nov 03 07:09:40.032346 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.032334 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabbdc26-a95a-401a-b787-023b23fd24e7" containerName="authorino" Nov 03 07:09:40.035421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.035403 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.092881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.092853 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 07:09:40.092881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.092869 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 07:09:40.142488 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.142460 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vdz8z\"" Nov 03 07:09:40.142488 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.142474 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 07:09:40.143874 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.143855 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-sckng"] Nov 03 07:09:40.157066 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.157028 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb400850-a2b2-4b94-839f-88e16402cf5f-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.157066 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.157059 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb400850-a2b2-4b94-839f-88e16402cf5f-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.157257 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.157095 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb400850-a2b2-4b94-839f-88e16402cf5f-home\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.157257 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.157190 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb400850-a2b2-4b94-839f-88e16402cf5f-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.157257 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.157217 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb400850-a2b2-4b94-839f-88e16402cf5f-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.157257 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.157241 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnbd7\" (UniqueName: \"kubernetes.io/projected/fb400850-a2b2-4b94-839f-88e16402cf5f-kube-api-access-rnbd7\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.258492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.258456 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb400850-a2b2-4b94-839f-88e16402cf5f-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.258492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.258491 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb400850-a2b2-4b94-839f-88e16402cf5f-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.258781 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.258526 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb400850-a2b2-4b94-839f-88e16402cf5f-home\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.258781 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.258659 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb400850-a2b2-4b94-839f-88e16402cf5f-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.258959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.258783 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb400850-a2b2-4b94-839f-88e16402cf5f-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.258959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.258820 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbd7\" (UniqueName: \"kubernetes.io/projected/fb400850-a2b2-4b94-839f-88e16402cf5f-kube-api-access-rnbd7\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.258959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.258936 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb400850-a2b2-4b94-839f-88e16402cf5f-home\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.259101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.259035 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb400850-a2b2-4b94-839f-88e16402cf5f-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.259101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.259092 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb400850-a2b2-4b94-839f-88e16402cf5f-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.261015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.260995 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb400850-a2b2-4b94-839f-88e16402cf5f-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.261176 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.261159 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb400850-a2b2-4b94-839f-88e16402cf5f-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.323661 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.323626 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnbd7\" (UniqueName: \"kubernetes.io/projected/fb400850-a2b2-4b94-839f-88e16402cf5f-kube-api-access-rnbd7\") pod \"echo-simulator-kserve-64b6f8f96c-sckng\" (UID: \"fb400850-a2b2-4b94-839f-88e16402cf5f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.349490 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.349450 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:40.735909 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:40.735828 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-sckng"] Nov 03 07:09:40.739115 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:09:40.739076 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb400850_a2b2_4b94_839f_88e16402cf5f.slice/crio-09badfba83dae45d6da76d7cd50f193ee489dc5e1eb266f3e772e2f79d0b2bc1 WatchSource:0}: Error finding container 09badfba83dae45d6da76d7cd50f193ee489dc5e1eb266f3e772e2f79d0b2bc1: Status 404 returned error can't find the container with id 09badfba83dae45d6da76d7cd50f193ee489dc5e1eb266f3e772e2f79d0b2bc1 Nov 03 07:09:41.497112 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:41.497074 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" event={"ID":"fb400850-a2b2-4b94-839f-88e16402cf5f","Type":"ContainerStarted","Data":"09badfba83dae45d6da76d7cd50f193ee489dc5e1eb266f3e772e2f79d0b2bc1"} Nov 03 07:09:43.507707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:43.507613 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" event={"ID":"fb400850-a2b2-4b94-839f-88e16402cf5f","Type":"ContainerStarted","Data":"0b3fdc71058d6a517b2548ad1da0e510651bf64e94790e952544ac7a1a189a59"} Nov 03 07:09:43.630002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:43.629938 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" podStartSLOduration=2.254128767 podStartE2EDuration="4.629920408s" totalImagesPullingTime="2.375791641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:09:43.629230534 +0000 UTC m=+638.821772297" watchObservedRunningTime="2026-11-03 07:09:43.629920408 +0000 UTC m=+638.822462159" Nov 03 07:09:44.349628 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.349594 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:44.497289 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.497260 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6"] Nov 03 07:09:44.500723 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.500693 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.599573 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.599545 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 07:09:44.599997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.599838 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b36e4ad-8060-4a62-b8c2-7468072396ae-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.599997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.599882 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b36e4ad-8060-4a62-b8c2-7468072396ae-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.600089 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.600030 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b36e4ad-8060-4a62-b8c2-7468072396ae-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.600089 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.600072 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b36e4ad-8060-4a62-b8c2-7468072396ae-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.600175 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.600089 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b36e4ad-8060-4a62-b8c2-7468072396ae-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.600175 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.600161 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgx7\" (UniqueName: \"kubernetes.io/projected/6b36e4ad-8060-4a62-b8c2-7468072396ae-kube-api-access-5fgx7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.605138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.605118 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6"] Nov 03 07:09:44.701465 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.701433 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b36e4ad-8060-4a62-b8c2-7468072396ae-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.701658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.701484 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b36e4ad-8060-4a62-b8c2-7468072396ae-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.701658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.701527 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b36e4ad-8060-4a62-b8c2-7468072396ae-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.701658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.701554 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgx7\" (UniqueName: \"kubernetes.io/projected/6b36e4ad-8060-4a62-b8c2-7468072396ae-kube-api-access-5fgx7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.701658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.701606 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b36e4ad-8060-4a62-b8c2-7468072396ae-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.701658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.701629 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b36e4ad-8060-4a62-b8c2-7468072396ae-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.702008 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.701976 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b36e4ad-8060-4a62-b8c2-7468072396ae-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.702146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.702046 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b36e4ad-8060-4a62-b8c2-7468072396ae-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.702146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.702068 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b36e4ad-8060-4a62-b8c2-7468072396ae-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.703925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.703897 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b36e4ad-8060-4a62-b8c2-7468072396ae-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.704159 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.704142 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b36e4ad-8060-4a62-b8c2-7468072396ae-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.767611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.767580 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgx7\" (UniqueName: \"kubernetes.io/projected/6b36e4ad-8060-4a62-b8c2-7468072396ae-kube-api-access-5fgx7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6\" (UID: \"6b36e4ad-8060-4a62-b8c2-7468072396ae\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:44.813726 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:44.813679 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:45.192938 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:45.192910 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6"] Nov 03 07:09:45.195257 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:09:45.195227 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b36e4ad_8060_4a62_b8c2_7468072396ae.slice/crio-60e13b0591430e78f57c5e8d3cfdbfb053fa23a9fe22cf93b3927b389e3bd577 WatchSource:0}: Error finding container 60e13b0591430e78f57c5e8d3cfdbfb053fa23a9fe22cf93b3927b389e3bd577: Status 404 returned error can't find the container with id 60e13b0591430e78f57c5e8d3cfdbfb053fa23a9fe22cf93b3927b389e3bd577 Nov 03 07:09:45.518308 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:45.518222 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" event={"ID":"6b36e4ad-8060-4a62-b8c2-7468072396ae","Type":"ContainerStarted","Data":"60e13b0591430e78f57c5e8d3cfdbfb053fa23a9fe22cf93b3927b389e3bd577"} Nov 03 07:09:47.529875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:47.529836 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" event={"ID":"6b36e4ad-8060-4a62-b8c2-7468072396ae","Type":"ContainerStarted","Data":"820053ee9c4605611c864ed042c6f04c1e589e1ecac9cef6f68efe0d033b4057"} Nov 03 07:09:47.651128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:47.651079 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" podStartSLOduration=2.181484769 podStartE2EDuration="3.651063216s" totalImagesPullingTime="1.469578447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:09:47.649739194 +0000 UTC m=+642.842280937" watchObservedRunningTime="2026-11-03 07:09:47.651063216 +0000 UTC m=+642.843604965" Nov 03 07:09:47.814436 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:47.814403 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:50.350147 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:50.350115 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:50.362801 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:50.362768 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:50.554195 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:50.554172 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-sckng" Nov 03 07:09:51.485042 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.482963 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz"] Nov 03 07:09:51.487757 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.487724 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.542531 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.542501 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz"] Nov 03 07:09:51.560920 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.560895 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37837181-c211-4404-846d-4b15159fac0c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.561076 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.560926 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37837181-c211-4404-846d-4b15159fac0c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.561076 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.561030 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37837181-c211-4404-846d-4b15159fac0c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.561163 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.561084 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37837181-c211-4404-846d-4b15159fac0c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.561163 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.561139 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcbk7\" (UniqueName: \"kubernetes.io/projected/37837181-c211-4404-846d-4b15159fac0c-kube-api-access-gcbk7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.561238 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.561171 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37837181-c211-4404-846d-4b15159fac0c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.590097 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.590075 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 07:09:51.662242 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.662211 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcbk7\" (UniqueName: \"kubernetes.io/projected/37837181-c211-4404-846d-4b15159fac0c-kube-api-access-gcbk7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.662387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.662258 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37837181-c211-4404-846d-4b15159fac0c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.662387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.662313 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37837181-c211-4404-846d-4b15159fac0c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.662387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.662339 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37837181-c211-4404-846d-4b15159fac0c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.662506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.662409 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37837181-c211-4404-846d-4b15159fac0c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.662506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.662451 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37837181-c211-4404-846d-4b15159fac0c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.662779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.662758 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37837181-c211-4404-846d-4b15159fac0c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.662830 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.662799 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37837181-c211-4404-846d-4b15159fac0c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.662926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.662907 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37837181-c211-4404-846d-4b15159fac0c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.664795 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.664776 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37837181-c211-4404-846d-4b15159fac0c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.664953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.664937 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37837181-c211-4404-846d-4b15159fac0c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.725444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.725417 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcbk7\" (UniqueName: \"kubernetes.io/projected/37837181-c211-4404-846d-4b15159fac0c-kube-api-access-gcbk7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz\" (UID: \"37837181-c211-4404-846d-4b15159fac0c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:51.800611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:51.800531 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:52.174793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:52.174773 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz"] Nov 03 07:09:52.177305 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:09:52.177276 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37837181_c211_4404_846d_4b15159fac0c.slice/crio-5c5c8a100eb13f9117cea8b60d536ba9c1ef69debf2088a66817792ed9a09329 WatchSource:0}: Error finding container 5c5c8a100eb13f9117cea8b60d536ba9c1ef69debf2088a66817792ed9a09329: Status 404 returned error can't find the container with id 5c5c8a100eb13f9117cea8b60d536ba9c1ef69debf2088a66817792ed9a09329 Nov 03 07:09:52.178669 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:52.178647 4409 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:09:52.552965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:52.552924 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" event={"ID":"37837181-c211-4404-846d-4b15159fac0c","Type":"ContainerStarted","Data":"5c5c8a100eb13f9117cea8b60d536ba9c1ef69debf2088a66817792ed9a09329"} Nov 03 07:09:53.558594 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:53.558559 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" event={"ID":"37837181-c211-4404-846d-4b15159fac0c","Type":"ContainerStarted","Data":"5889be73d3bbd9ade27321452114a3d5d4e518740c69c7264795df9da2f618cf"} Nov 03 07:09:53.801543 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:53.801500 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:09:54.814624 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:54.814595 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:54.827212 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:54.827196 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:54.949478 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:54.949433 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" podStartSLOduration=3.384670303 podStartE2EDuration="3.949419799s" totalImagesPullingTime="564.749496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:09:53.702174593 +0000 UTC m=+648.894716343" watchObservedRunningTime="2026-11-03 07:09:54.949419799 +0000 UTC m=+650.141961548" Nov 03 07:09:55.168940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.168868 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597"] Nov 03 07:09:55.172196 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.172176 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.226824 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.226789 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 07:09:55.232656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.232633 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597"] Nov 03 07:09:55.292851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.292827 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.292967 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.292855 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.292967 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.292877 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.293053 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.292994 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.293053 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.293014 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.293053 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.293031 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkcb\" (UniqueName: \"kubernetes.io/projected/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-kube-api-access-khkcb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.393927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.393897 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.394064 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.393935 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.394064 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.393955 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkcb\" (UniqueName: \"kubernetes.io/projected/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-kube-api-access-khkcb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.394064 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.394005 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.394064 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.394022 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.394064 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.394043 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.394319 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.394298 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.394385 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.394324 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.394440 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.394394 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.396401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.396383 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.396678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.396658 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.458206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.458147 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkcb\" (UniqueName: \"kubernetes.io/projected/cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1-kube-api-access-khkcb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wl597\" (UID: \"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.484884 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.484862 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:09:55.579147 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.579118 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" Nov 03 07:09:55.866344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:55.866319 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597"] Nov 03 07:09:55.868270 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:09:55.868242 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc3575a_55c9_4bb0_a6df_eb8ff497b5f1.slice/crio-7e7377ad5f4adebf7ba422a17fe2677280fcbce7d408767afe99c656b9d61537 WatchSource:0}: Error finding container 7e7377ad5f4adebf7ba422a17fe2677280fcbce7d408767afe99c656b9d61537: Status 404 returned error can't find the container with id 7e7377ad5f4adebf7ba422a17fe2677280fcbce7d408767afe99c656b9d61537 Nov 03 07:09:56.574416 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:56.574372 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" event={"ID":"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1","Type":"ContainerStarted","Data":"1e0406f756d5aa604e3d6580b671019b8eb35af431272e5d14ff13de4d36fb18"} Nov 03 07:09:56.574578 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:56.574473 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" event={"ID":"cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1","Type":"ContainerStarted","Data":"7e7377ad5f4adebf7ba422a17fe2677280fcbce7d408767afe99c656b9d61537"} Nov 03 07:09:56.695582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:56.695536 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" podStartSLOduration=1.1052391 podStartE2EDuration="1.695521098s" totalImagesPullingTime="590.281998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:09:56.694381734 +0000 UTC m=+651.886923485" watchObservedRunningTime="2026-11-03 07:09:56.695521098 +0000 UTC m=+651.888062848" Nov 03 07:09:57.485356 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:09:57.485330 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:10:01.801059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:01.801025 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:10:01.813258 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:01.813223 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:10:02.612099 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:02.612074 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz" Nov 03 07:10:05.485872 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:05.485844 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:10:05.498562 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:05.498540 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:10:05.624409 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:05.624384 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wl597" Nov 03 07:10:08.772772 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:10:08.772735 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:10:12.006050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.006014 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm"] Nov 03 07:10:12.010607 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.010591 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.034544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.034518 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.034651 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.034552 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.034651 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.034569 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.034651 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.034584 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.034823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.034706 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.034871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.034827 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.034871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.034849 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfh44\" (UniqueName: \"kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-kube-api-access-pfh44\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.034941 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.034903 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.034941 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.034936 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.066170 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.066143 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-89ggc\"" Nov 03 07:10:12.116512 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.116490 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm"] Nov 03 07:10:12.136086 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136061 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.136086 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136088 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfh44\" (UniqueName: \"kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-kube-api-access-pfh44\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.136235 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136111 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.136273 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136247 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.136312 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136292 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.136356 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136318 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.136356 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136341 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.136441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136366 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.136441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136413 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.136542 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136447 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.136678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136653 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.136826 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136753 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.136826 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.136787 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.137435 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.137410 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.138677 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.138660 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.138927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.138910 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.198887 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.198855 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.199053 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.199032 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfh44\" (UniqueName: \"kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-kube-api-access-pfh44\") pod \"maas-default-gateway-openshift-default-58d898bd55-x59tm\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.323509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.323480 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:12.751465 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:12.751443 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm"] Nov 03 07:10:12.753505 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:10:12.753480 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f4d0e8_3e95_44ca_a5d3_3109d2120453.slice/crio-c0dbd15eb9f0f8d95de7a5bec39aa952672a3c5b7dca287fe4103aff026fddbd WatchSource:0}: Error finding container c0dbd15eb9f0f8d95de7a5bec39aa952672a3c5b7dca287fe4103aff026fddbd: Status 404 returned error can't find the container with id c0dbd15eb9f0f8d95de7a5bec39aa952672a3c5b7dca287fe4103aff026fddbd Nov 03 07:10:13.647024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:13.646990 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" event={"ID":"68f4d0e8-3e95-44ca-a5d3-3109d2120453","Type":"ContainerStarted","Data":"c0dbd15eb9f0f8d95de7a5bec39aa952672a3c5b7dca287fe4103aff026fddbd"} Nov 03 07:10:16.481078 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:16.481051 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:10:16.482762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:16.481506 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:10:16.482762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:16.481630 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:10:16.663691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:16.663652 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" event={"ID":"68f4d0e8-3e95-44ca-a5d3-3109d2120453","Type":"ContainerStarted","Data":"428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c"} Nov 03 07:10:16.887494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:16.887439 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" podStartSLOduration=2.162596514 podStartE2EDuration="5.887426179s" totalImagesPullingTime="3.724829665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:10:16.837593704 +0000 UTC m=+672.030135453" watchObservedRunningTime="2026-11-03 07:10:16.887426179 +0000 UTC m=+672.079967930" Nov 03 07:10:17.323799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:17.323761 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:17.325180 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:17.325157 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-x59tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 07:10:17.325238 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:17.325205 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" podUID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 07:10:17.880739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:17.874219 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f56845b76-pf2td"] Nov 03 07:10:17.894155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:17.894117 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f56845b76-pf2td" Nov 03 07:10:17.933443 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:17.933419 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f56845b76-pf2td"] Nov 03 07:10:17.981173 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:17.981076 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 07:10:17.995092 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:17.995063 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9cm\" (UniqueName: \"kubernetes.io/projected/ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf-kube-api-access-2z9cm\") pod \"authorino-5f56845b76-pf2td\" (UID: \"ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf\") " pod="kuadrant-system/authorino-5f56845b76-pf2td" Nov 03 07:10:17.995242 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:17.995216 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf-tls-cert\") pod \"authorino-5f56845b76-pf2td\" (UID: \"ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf\") " pod="kuadrant-system/authorino-5f56845b76-pf2td" Nov 03 07:10:17.996918 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:17.996901 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 07:10:17.997021 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:17.996911 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zmffl\"" Nov 03 07:10:17.997021 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:17.996913 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 07:10:18.095993 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:18.095960 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9cm\" (UniqueName: \"kubernetes.io/projected/ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf-kube-api-access-2z9cm\") pod \"authorino-5f56845b76-pf2td\" (UID: \"ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf\") " pod="kuadrant-system/authorino-5f56845b76-pf2td" Nov 03 07:10:18.096154 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:18.096023 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf-tls-cert\") pod \"authorino-5f56845b76-pf2td\" (UID: \"ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf\") " pod="kuadrant-system/authorino-5f56845b76-pf2td" Nov 03 07:10:18.098487 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:18.098466 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf-tls-cert\") pod \"authorino-5f56845b76-pf2td\" (UID: \"ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf\") " pod="kuadrant-system/authorino-5f56845b76-pf2td" Nov 03 07:10:18.159385 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:18.159312 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9cm\" (UniqueName: \"kubernetes.io/projected/ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf-kube-api-access-2z9cm\") pod \"authorino-5f56845b76-pf2td\" (UID: \"ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf\") " pod="kuadrant-system/authorino-5f56845b76-pf2td" Nov 03 07:10:18.214899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:18.214871 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f56845b76-pf2td" Nov 03 07:10:18.323841 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:18.323808 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-x59tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 07:10:18.323988 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:18.323878 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" podUID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 07:10:18.616008 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:18.615976 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f56845b76-pf2td"] Nov 03 07:10:18.619118 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:10:18.619083 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee04ce1d_4bd5_4d30_880f_ae8c4a34ebcf.slice/crio-32051586fa52afc5f33f533e5637150e5ffd2dfdc0c1104f815356bb2b1c0f7e WatchSource:0}: Error finding container 32051586fa52afc5f33f533e5637150e5ffd2dfdc0c1104f815356bb2b1c0f7e: Status 404 returned error can't find the container with id 32051586fa52afc5f33f533e5637150e5ffd2dfdc0c1104f815356bb2b1c0f7e Nov 03 07:10:18.693615 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:18.693582 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f56845b76-pf2td" event={"ID":"ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf","Type":"ContainerStarted","Data":"32051586fa52afc5f33f533e5637150e5ffd2dfdc0c1104f815356bb2b1c0f7e"} Nov 03 07:10:19.324369 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:19.324332 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-x59tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 07:10:19.324861 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:19.324387 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" podUID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 07:10:20.324362 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:20.324329 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-x59tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 07:10:20.324524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:20.324376 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" podUID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 07:10:20.704123 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:20.704088 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f56845b76-pf2td" event={"ID":"ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf","Type":"ContainerStarted","Data":"57d565386fe6846ff4263845b2ca40beae42de7097cc0e4d74bcfd6358d24d04"} Nov 03 07:10:20.823195 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:20.823135 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f56845b76-pf2td" podStartSLOduration=2.303616316 podStartE2EDuration="3.823115022s" totalImagesPullingTime="1.519498706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:10:20.820952927 +0000 UTC m=+676.013494678" watchObservedRunningTime="2026-11-03 07:10:20.823115022 +0000 UTC m=+676.015656775" Nov 03 07:10:21.324493 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:21.324455 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-x59tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 07:10:21.324655 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:21.324520 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" podUID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 07:10:22.324734 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:22.324111 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-x59tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 07:10:22.324734 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:22.324197 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" podUID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 07:10:23.324127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:23.324092 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-x59tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 07:10:23.324299 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:23.324157 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" podUID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 07:10:24.752536 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:24.752501 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:24.753636 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:24.753612 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:24.754524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:24.754501 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:10:28.360747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:28.360691 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-hn2nn"] Nov 03 07:10:28.365812 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:28.365785 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" Nov 03 07:10:28.389728 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:28.389691 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knk4d\" (UniqueName: \"kubernetes.io/projected/aafe8fd2-50e9-4eea-8b6b-e686f1242127-kube-api-access-knk4d\") pod \"payload-processing-55fbbbf9dc-hn2nn\" (UID: \"aafe8fd2-50e9-4eea-8b6b-e686f1242127\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" Nov 03 07:10:28.389901 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:28.389806 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aafe8fd2-50e9-4eea-8b6b-e686f1242127-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-hn2nn\" (UID: \"aafe8fd2-50e9-4eea-8b6b-e686f1242127\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" Nov 03 07:10:28.423522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:28.423487 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-hn2nn"] Nov 03 07:10:28.470956 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:28.470926 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 07:10:28.470956 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:28.470928 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lztpq\"" Nov 03 07:10:28.491034 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:28.491003 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knk4d\" (UniqueName: \"kubernetes.io/projected/aafe8fd2-50e9-4eea-8b6b-e686f1242127-kube-api-access-knk4d\") pod \"payload-processing-55fbbbf9dc-hn2nn\" (UID: \"aafe8fd2-50e9-4eea-8b6b-e686f1242127\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" Nov 03 07:10:28.491208 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:28.491045 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aafe8fd2-50e9-4eea-8b6b-e686f1242127-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-hn2nn\" (UID: \"aafe8fd2-50e9-4eea-8b6b-e686f1242127\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" Nov 03 07:10:28.491811 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:28.491791 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aafe8fd2-50e9-4eea-8b6b-e686f1242127-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-hn2nn\" (UID: \"aafe8fd2-50e9-4eea-8b6b-e686f1242127\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" Nov 03 07:10:28.554737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:28.554691 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knk4d\" (UniqueName: \"kubernetes.io/projected/aafe8fd2-50e9-4eea-8b6b-e686f1242127-kube-api-access-knk4d\") pod \"payload-processing-55fbbbf9dc-hn2nn\" (UID: \"aafe8fd2-50e9-4eea-8b6b-e686f1242127\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" Nov 03 07:10:28.678466 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:28.678379 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" Nov 03 07:10:29.063749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:29.063705 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-hn2nn"] Nov 03 07:10:29.065940 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:10:29.065913 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafe8fd2_50e9_4eea_8b6b_e686f1242127.slice/crio-b29bc1246eaff8cde27b47e54a929525ce5db51dbd5c255336188398707540af WatchSource:0}: Error finding container b29bc1246eaff8cde27b47e54a929525ce5db51dbd5c255336188398707540af: Status 404 returned error can't find the container with id b29bc1246eaff8cde27b47e54a929525ce5db51dbd5c255336188398707540af Nov 03 07:10:29.749441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:29.749404 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" event={"ID":"aafe8fd2-50e9-4eea-8b6b-e686f1242127","Type":"ContainerStarted","Data":"b29bc1246eaff8cde27b47e54a929525ce5db51dbd5c255336188398707540af"} Nov 03 07:10:31.763797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:31.763757 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" event={"ID":"aafe8fd2-50e9-4eea-8b6b-e686f1242127","Type":"ContainerStarted","Data":"d2da640e46a04637191ada03f70b985ec4825a63a1fee1e55984bb64c82916dc"} Nov 03 07:10:31.764162 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:31.763906 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" Nov 03 07:10:31.882706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:31.882618 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" podStartSLOduration=1.386824064 podStartE2EDuration="3.882604644s" totalImagesPullingTime="2.49578058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:10:31.880849506 +0000 UTC m=+687.073391279" watchObservedRunningTime="2026-11-03 07:10:31.882604644 +0000 UTC m=+687.075146393" Nov 03 07:10:38.774061 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:10:38.774016 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:10:42.251556 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.251523 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr"] Nov 03 07:10:42.255335 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.255316 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.320045 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.320009 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3e173c-8bab-4805-8a40-fd77e4941d22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.320045 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.320050 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.320246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.320073 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.320246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.320176 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.320246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.320221 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.320344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.320303 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.320344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.320336 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.320411 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.320380 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fktdj\" (UniqueName: \"kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-kube-api-access-fktdj\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.320444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.320418 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.359475 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.359443 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr"] Nov 03 07:10:42.421187 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.421145 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.421382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.421199 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.421382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.421244 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fktdj\" (UniqueName: \"kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-kube-api-access-fktdj\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.421382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.421267 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.421382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.421288 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3e173c-8bab-4805-8a40-fd77e4941d22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.421382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.421314 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.421382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.421340 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.421382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.421370 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.421819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.421422 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.421879 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.421856 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.421935 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.421879 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.422052 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.422035 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.422126 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.422105 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.422496 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.422472 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3e173c-8bab-4805-8a40-fd77e4941d22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.423900 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.423875 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.424048 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.424032 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.484162 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.484126 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.484295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.484235 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fktdj\" (UniqueName: \"kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-kube-api-access-fktdj\") pod \"maas-default-gateway-openshift-default-58d898bd55-dk9tr\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.568427 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.568394 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:42.772791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.772756 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-hn2nn" Nov 03 07:10:42.997131 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:42.997108 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr"] Nov 03 07:10:42.999462 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:10:42.999437 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3e173c_8bab_4805_8a40_fd77e4941d22.slice/crio-14b444a67a6223e1c44301c930d2778e80f07802d106b065f6bfc2cc90b8b3ea WatchSource:0}: Error finding container 14b444a67a6223e1c44301c930d2778e80f07802d106b065f6bfc2cc90b8b3ea: Status 404 returned error can't find the container with id 14b444a67a6223e1c44301c930d2778e80f07802d106b065f6bfc2cc90b8b3ea Nov 03 07:10:43.001493 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:43.001444 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:10:43.001572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:43.001525 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:10:43.001572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:43.001555 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:10:43.820063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:43.820026 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" event={"ID":"6e3e173c-8bab-4805-8a40-fd77e4941d22","Type":"ContainerStarted","Data":"b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0"} Nov 03 07:10:43.820495 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:43.820086 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" event={"ID":"6e3e173c-8bab-4805-8a40-fd77e4941d22","Type":"ContainerStarted","Data":"14b444a67a6223e1c44301c930d2778e80f07802d106b065f6bfc2cc90b8b3ea"} Nov 03 07:10:43.949857 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:43.949788 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" podStartSLOduration=1.9497704850000002 podStartE2EDuration="1.949770485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:10:43.948846734 +0000 UTC m=+699.141388484" watchObservedRunningTime="2026-11-03 07:10:43.949770485 +0000 UTC m=+699.142312234" Nov 03 07:10:44.583739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:44.578538 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:44.583739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:44.578681 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 07:10:44.583739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:44.578752 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" podUID="6e3e173c-8bab-4805-8a40-fd77e4941d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 07:10:45.588738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:45.580796 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 07:10:45.588738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:45.580856 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" podUID="6e3e173c-8bab-4805-8a40-fd77e4941d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 07:10:46.583739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:46.579176 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 07:10:46.583739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:46.579232 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" podUID="6e3e173c-8bab-4805-8a40-fd77e4941d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 07:10:47.585747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:47.581737 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 07:10:47.585747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:47.581807 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" podUID="6e3e173c-8bab-4805-8a40-fd77e4941d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 07:10:48.577739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:48.574024 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 07:10:48.577739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:48.574145 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" podUID="6e3e173c-8bab-4805-8a40-fd77e4941d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 07:10:49.568778 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:49.568743 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-dk9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 07:10:49.569357 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:49.568830 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" podUID="6e3e173c-8bab-4805-8a40-fd77e4941d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 07:10:51.196063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:51.196030 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:51.196534 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:51.196428 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:10:51.197198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:10:51.197177 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:11:08.773061 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:11:08.773024 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:11:19.829916 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.829877 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p"] Nov 03 07:11:19.833376 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.833355 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.873961 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.873923 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.874160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.873970 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmwn\" (UniqueName: \"kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-kube-api-access-9dmwn\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.874160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.874111 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.874304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.874200 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.874304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.874229 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.874416 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.874301 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.874416 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.874323 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.874416 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.874402 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.874584 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.874452 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.915433 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.915403 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p"] Nov 03 07:11:19.966101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.966071 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-669d6b-openshift-default-dockercfg-f5qrn\"" Nov 03 07:11:19.979114 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.979070 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.979308 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.979183 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.979308 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.979279 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.979417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.979325 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmwn\" (UniqueName: \"kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-kube-api-access-9dmwn\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.979417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.979401 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.979525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.979483 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.979525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.979511 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.979621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.979573 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.979621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.979604 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.980913 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.980380 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.989214 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.981045 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.989544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.989501 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.989940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.989891 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.991181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.991089 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.992523 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.992497 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:19.993235 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:19.993180 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:20.053101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.053076 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:20.053309 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.053284 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmwn\" (UniqueName: \"kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-kube-api-access-9dmwn\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:20.147260 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.147154 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:20.631502 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.631480 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p"] Nov 03 07:11:20.633384 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:11:20.633360 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7f6705_eb50_49b1_bb3c_dcd0761f2c66.slice/crio-c753b455bdeadf81b1d2582f161562bad3e709bfde9f029d49c9db31c637eeac WatchSource:0}: Error finding container c753b455bdeadf81b1d2582f161562bad3e709bfde9f029d49c9db31c637eeac: Status 404 returned error can't find the container with id c753b455bdeadf81b1d2582f161562bad3e709bfde9f029d49c9db31c637eeac Nov 03 07:11:20.635708 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.635668 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:11:20.635828 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.635769 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:11:20.635828 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.635806 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:11:20.875384 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.875347 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv"] Nov 03 07:11:20.878942 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.878923 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:20.934130 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.934096 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6606d6-openshift-default-dockercfg-hxk2h\"" Nov 03 07:11:20.970056 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.970030 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv"] Nov 03 07:11:20.989307 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.989278 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:20.989480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.989326 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:20.989480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.989364 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:20.989611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.989512 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:20.989611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.989569 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:20.989611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.989596 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6sc4\" (UniqueName: \"kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-kube-api-access-z6sc4\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:20.989822 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.989733 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:20.989822 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.989789 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:20.989940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:20.989824 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.004920 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.004892 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" event={"ID":"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66","Type":"ContainerStarted","Data":"094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725"} Nov 03 07:11:21.005037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.004942 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" event={"ID":"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66","Type":"ContainerStarted","Data":"c753b455bdeadf81b1d2582f161562bad3e709bfde9f029d49c9db31c637eeac"} Nov 03 07:11:21.090645 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.090614 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.090843 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.090668 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.090843 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.090702 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.090843 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.090775 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.090843 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.090799 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6sc4\" (UniqueName: \"kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-kube-api-access-z6sc4\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.091066 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.090910 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.091066 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.090960 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.091066 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.090993 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.091066 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.091045 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.091259 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.091069 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.091259 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.091244 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.091502 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.091430 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.091602 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.091579 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.091838 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.091818 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.093771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.093459 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.093771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.093691 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.132881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.132831 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" podStartSLOduration=2.1328180899999998 podStartE2EDuration="2.13281809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:21.130550268 +0000 UTC m=+736.323092030" watchObservedRunningTime="2026-11-03 07:11:21.13281809 +0000 UTC m=+736.325359840" Nov 03 07:11:21.147536 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.147478 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:21.153808 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.153784 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:21.153989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.153965 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.154102 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.154085 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6sc4\" (UniqueName: \"kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-kube-api-access-z6sc4\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.192172 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.192151 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:21.219225 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.219197 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s"] Nov 03 07:11:21.222965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.222944 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.289739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.284943 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3fea3a-openshift-default-dockercfg-9452b\"" Nov 03 07:11:21.292790 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.292767 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s"] Nov 03 07:11:21.300696 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.300102 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.300696 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.300228 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt85x\" (UniqueName: \"kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-kube-api-access-nt85x\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.300696 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.300307 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.301065 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.301008 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.301065 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.301044 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.301065 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.301064 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbee391f-f81b-4975-aa4d-112fd30e5422-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.301226 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.301085 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.301226 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.301106 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.301226 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.301134 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.402611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.402529 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt85x\" (UniqueName: \"kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-kube-api-access-nt85x\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.402611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.402566 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.402888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.402702 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.402888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.402776 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.402888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.402804 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbee391f-f81b-4975-aa4d-112fd30e5422-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.402888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.402822 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.402888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.402846 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.403143 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.402892 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.403143 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.402956 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.403143 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.403053 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.403143 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.403060 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.403352 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.403300 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.403352 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.403334 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.403789 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.403765 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbee391f-f81b-4975-aa4d-112fd30e5422-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.405225 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.405210 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.405347 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.405331 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.465956 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.465934 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.466669 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.466647 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt85x\" (UniqueName: \"kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-kube-api-access-nt85x\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.540344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.540320 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:21.612291 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.612257 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv"] Nov 03 07:11:21.612942 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:11:21.612912 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8158bcb5_641e_40c7_9ac0_eb2d59f05235.slice/crio-9a8bc82cce734027678eb0d1e9e44f53134c98d7c1466cf396454086e224a06b WatchSource:0}: Error finding container 9a8bc82cce734027678eb0d1e9e44f53134c98d7c1466cf396454086e224a06b: Status 404 returned error can't find the container with id 9a8bc82cce734027678eb0d1e9e44f53134c98d7c1466cf396454086e224a06b Nov 03 07:11:21.616826 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.616759 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:11:21.618214 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.618031 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:11:21.618335 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.618313 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:11:21.939224 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.939194 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s"] Nov 03 07:11:21.940639 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:11:21.940612 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbee391f_f81b_4975_aa4d_112fd30e5422.slice/crio-64e7f713da4160d187016d370b0d354f381e27f260173ac07aad1b04db108db2 WatchSource:0}: Error finding container 64e7f713da4160d187016d370b0d354f381e27f260173ac07aad1b04db108db2: Status 404 returned error can't find the container with id 64e7f713da4160d187016d370b0d354f381e27f260173ac07aad1b04db108db2 Nov 03 07:11:21.943165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.943138 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:11:21.943244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.943196 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:11:21.943244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:21.943224 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:11:22.013823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:22.013792 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" event={"ID":"fbee391f-f81b-4975-aa4d-112fd30e5422","Type":"ContainerStarted","Data":"fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43"} Nov 03 07:11:22.013930 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:22.013849 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" event={"ID":"fbee391f-f81b-4975-aa4d-112fd30e5422","Type":"ContainerStarted","Data":"64e7f713da4160d187016d370b0d354f381e27f260173ac07aad1b04db108db2"} Nov 03 07:11:22.015892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:22.015865 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" event={"ID":"8158bcb5-641e-40c7-9ac0-eb2d59f05235","Type":"ContainerStarted","Data":"c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6"} Nov 03 07:11:22.016003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:22.015916 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" event={"ID":"8158bcb5-641e-40c7-9ac0-eb2d59f05235","Type":"ContainerStarted","Data":"9a8bc82cce734027678eb0d1e9e44f53134c98d7c1466cf396454086e224a06b"} Nov 03 07:11:22.016612 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:22.016589 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:22.017553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:22.017537 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:11:22.138597 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:22.138553 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" podStartSLOduration=1.138540071 podStartE2EDuration="1.138540071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:22.136846815 +0000 UTC m=+737.329388566" watchObservedRunningTime="2026-11-03 07:11:22.138540071 +0000 UTC m=+737.331081862" Nov 03 07:11:22.193309 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:22.193234 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:22.198663 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:22.198641 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:22.397923 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:22.397871 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" podStartSLOduration=2.397854676 podStartE2EDuration="2.397854676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:22.39590673 +0000 UTC m=+737.588448480" watchObservedRunningTime="2026-11-03 07:11:22.397854676 +0000 UTC m=+737.590396473" Nov 03 07:11:22.545964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:22.541492 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:23.020741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:23.020679 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:23.021560 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:23.021545 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:11:23.546585 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:23.546558 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:24.025033 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:24.024995 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:24.025850 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:24.025833 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:11:29.264643 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.264606 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59"] Nov 03 07:11:29.268387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.268370 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" Nov 03 07:11:29.325879 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.325856 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-9697f7\"" Nov 03 07:11:29.327388 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.327364 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59"] Nov 03 07:11:29.371919 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.371873 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-9697f7-dockercfg-7sxt4\"" Nov 03 07:11:29.389031 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.388989 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34efdb29-aee9-40f9-bb8e-92410e7b9e48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59\" (UID: \"34efdb29-aee9-40f9-bb8e-92410e7b9e48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" Nov 03 07:11:29.389229 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.389122 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfg2d\" (UniqueName: \"kubernetes.io/projected/34efdb29-aee9-40f9-bb8e-92410e7b9e48-kube-api-access-mfg2d\") pod \"payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59\" (UID: \"34efdb29-aee9-40f9-bb8e-92410e7b9e48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" Nov 03 07:11:29.392077 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.392052 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw"] Nov 03 07:11:29.397407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.396140 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" Nov 03 07:11:29.454057 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.454029 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw"] Nov 03 07:11:29.490274 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.490251 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxfgc\" (UniqueName: \"kubernetes.io/projected/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-kube-api-access-gxfgc\") pod \"payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw\" (UID: \"9f03bf4c-6138-4f5a-a3c7-739cb83219bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" Nov 03 07:11:29.490404 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.490290 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfg2d\" (UniqueName: \"kubernetes.io/projected/34efdb29-aee9-40f9-bb8e-92410e7b9e48-kube-api-access-mfg2d\") pod \"payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59\" (UID: \"34efdb29-aee9-40f9-bb8e-92410e7b9e48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" Nov 03 07:11:29.490447 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.490434 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw\" (UID: \"9f03bf4c-6138-4f5a-a3c7-739cb83219bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" Nov 03 07:11:29.490503 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.490489 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34efdb29-aee9-40f9-bb8e-92410e7b9e48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59\" (UID: \"34efdb29-aee9-40f9-bb8e-92410e7b9e48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" Nov 03 07:11:29.491237 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.491220 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34efdb29-aee9-40f9-bb8e-92410e7b9e48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59\" (UID: \"34efdb29-aee9-40f9-bb8e-92410e7b9e48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" Nov 03 07:11:29.553486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.553461 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfg2d\" (UniqueName: \"kubernetes.io/projected/34efdb29-aee9-40f9-bb8e-92410e7b9e48-kube-api-access-mfg2d\") pod \"payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59\" (UID: \"34efdb29-aee9-40f9-bb8e-92410e7b9e48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" Nov 03 07:11:29.581984 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.581962 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" Nov 03 07:11:29.591282 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.591259 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxfgc\" (UniqueName: \"kubernetes.io/projected/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-kube-api-access-gxfgc\") pod \"payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw\" (UID: \"9f03bf4c-6138-4f5a-a3c7-739cb83219bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" Nov 03 07:11:29.591367 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.591318 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw\" (UID: \"9f03bf4c-6138-4f5a-a3c7-739cb83219bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" Nov 03 07:11:29.591943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.591925 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw\" (UID: \"9f03bf4c-6138-4f5a-a3c7-739cb83219bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" Nov 03 07:11:29.654819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.654786 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxfgc\" (UniqueName: \"kubernetes.io/projected/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-kube-api-access-gxfgc\") pod \"payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw\" (UID: \"9f03bf4c-6138-4f5a-a3c7-739cb83219bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" Nov 03 07:11:29.711667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.711642 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" Nov 03 07:11:29.956366 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:29.956339 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59"] Nov 03 07:11:29.958729 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:11:29.958687 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34efdb29_aee9_40f9_bb8e_92410e7b9e48.slice/crio-6d3d109251df5dd9943fe6a25e9ccd9efb9f7cbc7db092d0004f63b8081f0338 WatchSource:0}: Error finding container 6d3d109251df5dd9943fe6a25e9ccd9efb9f7cbc7db092d0004f63b8081f0338: Status 404 returned error can't find the container with id 6d3d109251df5dd9943fe6a25e9ccd9efb9f7cbc7db092d0004f63b8081f0338 Nov 03 07:11:30.055011 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.054979 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" event={"ID":"34efdb29-aee9-40f9-bb8e-92410e7b9e48","Type":"ContainerStarted","Data":"cfc21acad3b412e5eea8235e5cd5c4d3159f81a9bed5296f9b6f5c0c94bb8af0"} Nov 03 07:11:30.055133 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.055036 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" event={"ID":"34efdb29-aee9-40f9-bb8e-92410e7b9e48","Type":"ContainerStarted","Data":"6d3d109251df5dd9943fe6a25e9ccd9efb9f7cbc7db092d0004f63b8081f0338"} Nov 03 07:11:30.055133 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.055078 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" Nov 03 07:11:30.096473 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.096442 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw"] Nov 03 07:11:30.098647 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:11:30.098614 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f03bf4c_6138_4f5a_a3c7_739cb83219bb.slice/crio-337e9cc704f1a118a8012deba72c27c55970d10fc9dbbf367c6f30053d9890de WatchSource:0}: Error finding container 337e9cc704f1a118a8012deba72c27c55970d10fc9dbbf367c6f30053d9890de: Status 404 returned error can't find the container with id 337e9cc704f1a118a8012deba72c27c55970d10fc9dbbf367c6f30053d9890de Nov 03 07:11:30.176549 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.176508 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" podStartSLOduration=1.176493953 podStartE2EDuration="1.176493953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:30.176130919 +0000 UTC m=+745.368672658" watchObservedRunningTime="2026-11-03 07:11:30.176493953 +0000 UTC m=+745.369035719" Nov 03 07:11:30.541158 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.541127 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh"] Nov 03 07:11:30.544866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.544845 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:30.597959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.597941 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-669d6b-serving-cert\"" Nov 03 07:11:30.600418 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.600401 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-maas-api-tls\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:30.600564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.600547 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c28244-4715-46d9-9162-aeaa1d92dc1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:30.600613 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.600602 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-metrics-certs\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:30.600652 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.600622 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwxs7\" (UniqueName: \"kubernetes.io/projected/79c28244-4715-46d9-9162-aeaa1d92dc1e-kube-api-access-pwxs7\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:30.606146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.606121 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh"] Nov 03 07:11:30.701682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.701654 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c28244-4715-46d9-9162-aeaa1d92dc1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:30.701845 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.701707 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-metrics-certs\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:30.701845 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.701753 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwxs7\" (UniqueName: \"kubernetes.io/projected/79c28244-4715-46d9-9162-aeaa1d92dc1e-kube-api-access-pwxs7\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:30.701845 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.701778 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-maas-api-tls\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:30.701984 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:11:30.701928 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-serving-cert: secret "maas-api-e2e-worker-w2-669d6b-serving-cert" not found Nov 03 07:11:30.702038 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:11:30.702000 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-maas-api-tls podName:79c28244-4715-46d9-9162-aeaa1d92dc1e nodeName:}" failed. No retries permitted until 2026-11-03 07:11:31.201980638 +0000 UTC m=+746.394522383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-maas-api-tls") pod "maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" (UID: "79c28244-4715-46d9-9162-aeaa1d92dc1e") : secret "maas-api-e2e-worker-w2-669d6b-serving-cert" not found Nov 03 07:11:30.702354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.702334 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c28244-4715-46d9-9162-aeaa1d92dc1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:30.704353 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.704329 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-metrics-certs\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:30.767105 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:30.767076 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwxs7\" (UniqueName: \"kubernetes.io/projected/79c28244-4715-46d9-9162-aeaa1d92dc1e-kube-api-access-pwxs7\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:31.060892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:31.060854 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" event={"ID":"9f03bf4c-6138-4f5a-a3c7-739cb83219bb","Type":"ContainerStarted","Data":"b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6"} Nov 03 07:11:31.061244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:31.060909 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" event={"ID":"9f03bf4c-6138-4f5a-a3c7-739cb83219bb","Type":"ContainerStarted","Data":"337e9cc704f1a118a8012deba72c27c55970d10fc9dbbf367c6f30053d9890de"} Nov 03 07:11:31.061244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:31.061057 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" Nov 03 07:11:31.181059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:31.181007 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" podStartSLOduration=2.180991488 podStartE2EDuration="2.180991488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:31.17932354 +0000 UTC m=+746.371865291" watchObservedRunningTime="2026-11-03 07:11:31.180991488 +0000 UTC m=+746.373533239" Nov 03 07:11:31.208085 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:31.208050 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-maas-api-tls\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:31.210589 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:31.210566 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-maas-api-tls\") pod \"maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:31.458016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:31.457926 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:31.835053 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:31.835015 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh"] Nov 03 07:11:31.842183 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:11:31.842158 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c28244_4715_46d9_9162_aeaa1d92dc1e.slice/crio-7710bc375bcbee503379ad94cd3feecbc5c35e9e2d51ad8ac875660c51e91a50 WatchSource:0}: Error finding container 7710bc375bcbee503379ad94cd3feecbc5c35e9e2d51ad8ac875660c51e91a50: Status 404 returned error can't find the container with id 7710bc375bcbee503379ad94cd3feecbc5c35e9e2d51ad8ac875660c51e91a50 Nov 03 07:11:32.066532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:32.066505 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" event={"ID":"79c28244-4715-46d9-9162-aeaa1d92dc1e","Type":"ContainerStarted","Data":"7710bc375bcbee503379ad94cd3feecbc5c35e9e2d51ad8ac875660c51e91a50"} Nov 03 07:11:33.072701 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:33.072668 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" event={"ID":"79c28244-4715-46d9-9162-aeaa1d92dc1e","Type":"ContainerStarted","Data":"b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035"} Nov 03 07:11:33.073107 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:33.072937 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:33.194174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:33.194085 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" podStartSLOduration=2.145303455 podStartE2EDuration="3.194072005s" totalImagesPullingTime="1.04876855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:33.192637147 +0000 UTC m=+748.385178897" watchObservedRunningTime="2026-11-03 07:11:33.194072005 +0000 UTC m=+748.386613754" Nov 03 07:11:36.013742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.013698 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv"] Nov 03 07:11:36.017369 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.017353 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.071682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.071660 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-ffee69-serving-cert\"" Nov 03 07:11:36.122229 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.122203 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv"] Nov 03 07:11:36.165737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.165054 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e68f6-d602-4c83-80a1-0323c92a6f98-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.165737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.165231 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-metrics-certs\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.165737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.165267 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.165737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.165303 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbq99\" (UniqueName: \"kubernetes.io/projected/3e6e68f6-d602-4c83-80a1-0323c92a6f98-kube-api-access-vbq99\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.268273 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.268177 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbq99\" (UniqueName: \"kubernetes.io/projected/3e6e68f6-d602-4c83-80a1-0323c92a6f98-kube-api-access-vbq99\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.268435 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.268338 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e68f6-d602-4c83-80a1-0323c92a6f98-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.269185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.269142 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e68f6-d602-4c83-80a1-0323c92a6f98-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.269613 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.269588 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-metrics-certs\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.279923 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.269925 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.279923 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:11:36.270102 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-serving-cert: secret "maas-api-e2e-worker-w4-ffee69-serving-cert" not found Nov 03 07:11:36.279923 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:11:36.270162 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-maas-api-tls podName:3e6e68f6-d602-4c83-80a1-0323c92a6f98 nodeName:}" failed. No retries permitted until 2026-11-03 07:11:36.770143256 +0000 UTC m=+751.962684998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-maas-api-tls") pod "maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" (UID: "3e6e68f6-d602-4c83-80a1-0323c92a6f98") : secret "maas-api-e2e-worker-w4-ffee69-serving-cert" not found Nov 03 07:11:36.283747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.281205 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-metrics-certs\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.335955 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.335917 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbq99\" (UniqueName: \"kubernetes.io/projected/3e6e68f6-d602-4c83-80a1-0323c92a6f98-kube-api-access-vbq99\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.775076 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.775047 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.777551 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.777521 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:36.934978 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:36.934943 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:37.345186 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:37.345164 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv"] Nov 03 07:11:37.347296 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:11:37.347264 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6e68f6_d602_4c83_80a1_0323c92a6f98.slice/crio-9f73bc06a7e2c9b0396d40d91bcd1615a5e8564585904d381dce25e1a75131ba WatchSource:0}: Error finding container 9f73bc06a7e2c9b0396d40d91bcd1615a5e8564585904d381dce25e1a75131ba: Status 404 returned error can't find the container with id 9f73bc06a7e2c9b0396d40d91bcd1615a5e8564585904d381dce25e1a75131ba Nov 03 07:11:38.099068 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:38.099029 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" event={"ID":"3e6e68f6-d602-4c83-80a1-0323c92a6f98","Type":"ContainerStarted","Data":"9f73bc06a7e2c9b0396d40d91bcd1615a5e8564585904d381dce25e1a75131ba"} Nov 03 07:11:38.774865 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:11:38.774829 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:11:39.082786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:39.082761 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:11:39.109483 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:39.109444 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" event={"ID":"3e6e68f6-d602-4c83-80a1-0323c92a6f98","Type":"ContainerStarted","Data":"aaf825570abf3c7da667c2b84efe6db4c3343fd3c54311a7b0c2994004e52eb6"} Nov 03 07:11:39.109673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:39.109600 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:39.336130 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:39.336020 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" podStartSLOduration=3.432561505 podStartE2EDuration="4.336001172s" totalImagesPullingTime="903.439667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:39.333767718 +0000 UTC m=+754.526309465" watchObservedRunningTime="2026-11-03 07:11:39.336001172 +0000 UTC m=+754.528542923" Nov 03 07:11:41.064851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:41.064820 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" Nov 03 07:11:41.829546 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:41.829509 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q"] Nov 03 07:11:41.839775 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:41.839746 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" Nov 03 07:11:41.891595 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:41.891563 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q"] Nov 03 07:11:41.894357 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:41.894330 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-6606d6\"" Nov 03 07:11:41.942557 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:41.942520 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-6606d6-dockercfg-hdqjw\"" Nov 03 07:11:41.943308 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:41.943282 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwt5\" (UniqueName: \"kubernetes.io/projected/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-kube-api-access-qtwt5\") pod \"payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q\" (UID: \"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" Nov 03 07:11:41.943449 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:41.943331 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q\" (UID: \"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" Nov 03 07:11:42.044630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:42.044596 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwt5\" (UniqueName: \"kubernetes.io/projected/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-kube-api-access-qtwt5\") pod \"payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q\" (UID: \"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" Nov 03 07:11:42.044630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:42.044636 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q\" (UID: \"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" Nov 03 07:11:42.045292 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:42.045272 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q\" (UID: \"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" Nov 03 07:11:42.069662 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:42.069637 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" Nov 03 07:11:42.107570 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:42.107503 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwt5\" (UniqueName: \"kubernetes.io/projected/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-kube-api-access-qtwt5\") pod \"payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q\" (UID: \"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" Nov 03 07:11:42.154503 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:42.154465 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" Nov 03 07:11:42.534173 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:42.534146 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q"] Nov 03 07:11:42.536998 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:11:42.536969 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bc426d_4cee_46ca_b9fc_f20f463fb6aa.slice/crio-f034aa4fe0005f2225af2dd4661d8a94f47ad0b9183e10f0de8e89df04e3490e WatchSource:0}: Error finding container f034aa4fe0005f2225af2dd4661d8a94f47ad0b9183e10f0de8e89df04e3490e: Status 404 returned error can't find the container with id f034aa4fe0005f2225af2dd4661d8a94f47ad0b9183e10f0de8e89df04e3490e Nov 03 07:11:43.143174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:43.143137 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" event={"ID":"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa","Type":"ContainerStarted","Data":"b33db81788f7f11ad927ebc54cd709bf1e24d1dd7e73c7178bca01c698eae4fe"} Nov 03 07:11:43.143550 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:43.143201 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" event={"ID":"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa","Type":"ContainerStarted","Data":"f034aa4fe0005f2225af2dd4661d8a94f47ad0b9183e10f0de8e89df04e3490e"} Nov 03 07:11:43.143550 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:43.143227 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" Nov 03 07:11:43.266248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:43.266196 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" podStartSLOduration=2.26617924 podStartE2EDuration="2.26617924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:43.264126147 +0000 UTC m=+758.456667896" watchObservedRunningTime="2026-11-03 07:11:43.26617924 +0000 UTC m=+758.458720990" Nov 03 07:11:45.120633 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:45.120606 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:11:46.811411 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:46.811319 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6"] Nov 03 07:11:46.815125 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:46.815103 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" Nov 03 07:11:46.873091 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:46.873029 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ffee69\"" Nov 03 07:11:46.902386 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:46.902351 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963a8556-7780-49ec-84bf-de9a09ff4065-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6\" (UID: \"963a8556-7780-49ec-84bf-de9a09ff4065\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" Nov 03 07:11:46.902545 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:46.902524 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvwrw\" (UniqueName: \"kubernetes.io/projected/963a8556-7780-49ec-84bf-de9a09ff4065-kube-api-access-tvwrw\") pod \"payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6\" (UID: \"963a8556-7780-49ec-84bf-de9a09ff4065\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" Nov 03 07:11:46.934478 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:46.934443 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6"] Nov 03 07:11:46.934930 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:46.934837 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ffee69-dockercfg-bhn7b\"" Nov 03 07:11:47.003393 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:47.003358 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963a8556-7780-49ec-84bf-de9a09ff4065-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6\" (UID: \"963a8556-7780-49ec-84bf-de9a09ff4065\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" Nov 03 07:11:47.003562 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:47.003507 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvwrw\" (UniqueName: \"kubernetes.io/projected/963a8556-7780-49ec-84bf-de9a09ff4065-kube-api-access-tvwrw\") pod \"payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6\" (UID: \"963a8556-7780-49ec-84bf-de9a09ff4065\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" Nov 03 07:11:47.004114 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:47.004094 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963a8556-7780-49ec-84bf-de9a09ff4065-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6\" (UID: \"963a8556-7780-49ec-84bf-de9a09ff4065\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" Nov 03 07:11:47.086380 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:47.086323 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvwrw\" (UniqueName: \"kubernetes.io/projected/963a8556-7780-49ec-84bf-de9a09ff4065-kube-api-access-tvwrw\") pod \"payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6\" (UID: \"963a8556-7780-49ec-84bf-de9a09ff4065\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" Nov 03 07:11:47.128522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:47.128494 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" Nov 03 07:11:47.509736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:47.509694 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6"] Nov 03 07:11:47.511837 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:11:47.511809 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963a8556_7780_49ec_84bf_de9a09ff4065.slice/crio-fb220c2c8147e3f5b3d5829393b0c7191f35af22e74a216d75dabf4a465d99c0 WatchSource:0}: Error finding container fb220c2c8147e3f5b3d5829393b0c7191f35af22e74a216d75dabf4a465d99c0: Status 404 returned error can't find the container with id fb220c2c8147e3f5b3d5829393b0c7191f35af22e74a216d75dabf4a465d99c0 Nov 03 07:11:48.169348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:48.169317 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" event={"ID":"963a8556-7780-49ec-84bf-de9a09ff4065","Type":"ContainerStarted","Data":"153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d"} Nov 03 07:11:48.169820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:48.169363 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" event={"ID":"963a8556-7780-49ec-84bf-de9a09ff4065","Type":"ContainerStarted","Data":"fb220c2c8147e3f5b3d5829393b0c7191f35af22e74a216d75dabf4a465d99c0"} Nov 03 07:11:48.169820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:48.169436 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" Nov 03 07:11:48.289645 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:48.289588 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" podStartSLOduration=2.289568203 podStartE2EDuration="2.289568203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:48.289208778 +0000 UTC m=+763.481750528" watchObservedRunningTime="2026-11-03 07:11:48.289568203 +0000 UTC m=+763.482109955" Nov 03 07:11:52.362214 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.362176 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp"] Nov 03 07:11:52.366479 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.366458 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" Nov 03 07:11:52.423949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.423923 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp"] Nov 03 07:11:52.470459 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.470429 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-3b41b9\"" Nov 03 07:11:52.470584 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.470470 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-3b41b9-dockercfg-b5wqc\"" Nov 03 07:11:52.489339 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.489315 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx8fb\" (UniqueName: \"kubernetes.io/projected/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-kube-api-access-jx8fb\") pod \"payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp\" (UID: \"f3c5b6de-bfce-4501-b6aa-44fb2359ae80\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" Nov 03 07:11:52.489462 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.489437 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp\" (UID: \"f3c5b6de-bfce-4501-b6aa-44fb2359ae80\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" Nov 03 07:11:52.529561 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.529537 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58"] Nov 03 07:11:52.533294 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.533278 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:52.587325 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.587302 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-3fea3a-serving-cert\"" Nov 03 07:11:52.590861 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.590837 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-metrics-certs\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:52.590949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.590894 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8fb\" (UniqueName: \"kubernetes.io/projected/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-kube-api-access-jx8fb\") pod \"payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp\" (UID: \"f3c5b6de-bfce-4501-b6aa-44fb2359ae80\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" Nov 03 07:11:52.591011 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.590948 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:52.591112 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.591095 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41668674-15da-4311-a5fb-4254db00797e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:52.591175 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.591166 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhhmn\" (UniqueName: \"kubernetes.io/projected/41668674-15da-4311-a5fb-4254db00797e-kube-api-access-rhhmn\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:52.591228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.591207 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp\" (UID: \"f3c5b6de-bfce-4501-b6aa-44fb2359ae80\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" Nov 03 07:11:52.592020 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.591997 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp\" (UID: \"f3c5b6de-bfce-4501-b6aa-44fb2359ae80\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" Nov 03 07:11:52.594533 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.594509 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58"] Nov 03 07:11:52.660808 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.660752 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8fb\" (UniqueName: \"kubernetes.io/projected/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-kube-api-access-jx8fb\") pod \"payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp\" (UID: \"f3c5b6de-bfce-4501-b6aa-44fb2359ae80\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" Nov 03 07:11:52.680297 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.680271 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" Nov 03 07:11:52.691987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.691969 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-metrics-certs\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:52.692076 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.692040 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:52.692148 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.692131 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41668674-15da-4311-a5fb-4254db00797e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:52.692374 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:11:52.692196 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-serving-cert: secret "maas-api-e2e-worker-w5-3fea3a-serving-cert" not found Nov 03 07:11:52.692374 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:11:52.692288 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-maas-api-tls podName:41668674-15da-4311-a5fb-4254db00797e nodeName:}" failed. No retries permitted until 2026-11-03 07:11:53.192264036 +0000 UTC m=+768.384805781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-maas-api-tls") pod "maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" (UID: "41668674-15da-4311-a5fb-4254db00797e") : secret "maas-api-e2e-worker-w5-3fea3a-serving-cert" not found Nov 03 07:11:52.692820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.692789 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhhmn\" (UniqueName: \"kubernetes.io/projected/41668674-15da-4311-a5fb-4254db00797e-kube-api-access-rhhmn\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:52.693064 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.693014 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41668674-15da-4311-a5fb-4254db00797e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:52.695024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.694999 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-metrics-certs\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:52.756001 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:52.755977 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhhmn\" (UniqueName: \"kubernetes.io/projected/41668674-15da-4311-a5fb-4254db00797e-kube-api-access-rhhmn\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:53.068626 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:53.068601 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp"] Nov 03 07:11:53.070507 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:11:53.070483 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c5b6de_bfce_4501_b6aa_44fb2359ae80.slice/crio-f95972177c14f268e939af9ede1e9cbc97ab19546e71a45a853926421785f008 WatchSource:0}: Error finding container f95972177c14f268e939af9ede1e9cbc97ab19546e71a45a853926421785f008: Status 404 returned error can't find the container with id f95972177c14f268e939af9ede1e9cbc97ab19546e71a45a853926421785f008 Nov 03 07:11:53.195916 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:53.195885 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" event={"ID":"f3c5b6de-bfce-4501-b6aa-44fb2359ae80","Type":"ContainerStarted","Data":"625a296bfa749bb61b6ff360d918bb7690f89260803e3b49817b93697a7353de"} Nov 03 07:11:53.196047 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:53.195937 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" event={"ID":"f3c5b6de-bfce-4501-b6aa-44fb2359ae80","Type":"ContainerStarted","Data":"f95972177c14f268e939af9ede1e9cbc97ab19546e71a45a853926421785f008"} Nov 03 07:11:53.196047 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:53.196014 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" Nov 03 07:11:53.216090 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:53.216068 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:53.218359 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:53.218339 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:53.316418 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:53.316375 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" podStartSLOduration=1.316365245 podStartE2EDuration="1.316365245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:53.313660546 +0000 UTC m=+768.506202296" watchObservedRunningTime="2026-11-03 07:11:53.316365245 +0000 UTC m=+768.508906995" Nov 03 07:11:53.447098 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:53.446351 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:53.873164 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:53.873139 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58"] Nov 03 07:11:53.875241 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:11:53.875217 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41668674_15da_4311_a5fb_4254db00797e.slice/crio-9d15aa5e601b6791b2dcb371512273b71a4d8f0a5152b188d649e52a5bffbb3f WatchSource:0}: Error finding container 9d15aa5e601b6791b2dcb371512273b71a4d8f0a5152b188d649e52a5bffbb3f: Status 404 returned error can't find the container with id 9d15aa5e601b6791b2dcb371512273b71a4d8f0a5152b188d649e52a5bffbb3f Nov 03 07:11:54.151033 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:54.150968 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" Nov 03 07:11:54.202005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:54.201975 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" event={"ID":"41668674-15da-4311-a5fb-4254db00797e","Type":"ContainerStarted","Data":"9d15aa5e601b6791b2dcb371512273b71a4d8f0a5152b188d649e52a5bffbb3f"} Nov 03 07:11:55.208589 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:55.208552 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" event={"ID":"41668674-15da-4311-a5fb-4254db00797e","Type":"ContainerStarted","Data":"3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19"} Nov 03 07:11:55.209037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:55.208629 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:11:55.332051 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:55.332000 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" podStartSLOduration=2.556352237 podStartE2EDuration="3.331985778s" totalImagesPullingTime="775.633541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:11:55.330144968 +0000 UTC m=+770.522686731" watchObservedRunningTime="2026-11-03 07:11:55.331985778 +0000 UTC m=+770.524527527" Nov 03 07:11:59.190773 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:11:59.190745 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" Nov 03 07:12:01.218669 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:01.218639 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:12:04.205394 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:04.205364 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" Nov 03 07:12:06.266500 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:06.266413 4409 generic.go:356] "Generic (PLEG): container finished" podID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerID="c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6" exitCode=0 Nov 03 07:12:06.266500 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:06.266487 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" event={"ID":"8158bcb5-641e-40c7-9ac0-eb2d59f05235","Type":"ContainerDied","Data":"c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6"} Nov 03 07:12:06.266971 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:06.266839 4409 scope.go:122] "RemoveContainer" containerID="c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6" Nov 03 07:12:06.268099 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:06.268073 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:12:06.268159 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:06.268145 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:12:06.268196 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:06.268184 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:12:07.304750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:07.304455 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" event={"ID":"8158bcb5-641e-40c7-9ac0-eb2d59f05235","Type":"ContainerStarted","Data":"e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6"} Nov 03 07:12:08.021006 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:08.020856 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:12:08.195936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:08.195878 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:12:08.197920 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:08.197864 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 07:12:08.198176 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:08.198125 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 07:12:08.817300 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:12:08.817213 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:12:09.198795 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:09.196049 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 07:12:09.198795 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:09.196116 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 07:12:10.197295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:10.197262 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 07:12:10.197903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:10.197871 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 07:12:11.197738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:11.197094 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 07:12:11.197738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:11.197160 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 07:12:12.193128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:12.193088 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 07:12:12.193435 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:12.193403 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 07:12:13.197284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:13.197256 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:12:13.198417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:13.198399 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:12:16.366176 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:16.366143 4409 generic.go:356] "Generic (PLEG): container finished" podID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerID="fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43" exitCode=0 Nov 03 07:12:16.366788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:16.366228 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" event={"ID":"fbee391f-f81b-4975-aa4d-112fd30e5422","Type":"ContainerDied","Data":"fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43"} Nov 03 07:12:16.366788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:16.366640 4409 scope.go:122] "RemoveContainer" containerID="fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43" Nov 03 07:12:16.367927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:16.367896 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:12:16.368041 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:16.367966 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:12:16.368041 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:16.367999 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:12:17.399340 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:17.399301 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" event={"ID":"fbee391f-f81b-4975-aa4d-112fd30e5422","Type":"ContainerStarted","Data":"33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67"} Nov 03 07:12:17.541225 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:17.541196 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:12:17.541466 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:17.541446 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 07:12:17.541531 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:17.541494 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 07:12:18.541642 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:18.541609 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 07:12:18.542131 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:18.541669 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 07:12:19.541265 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:19.541233 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 07:12:19.541441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:19.541288 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 07:12:20.540794 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:20.540763 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 07:12:20.541170 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:20.540828 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 07:12:21.541359 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:21.541325 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 07:12:21.541804 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:21.541374 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 07:12:22.619033 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:22.619000 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:12:22.619864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:22.619842 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:12:22.623230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:22.623207 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:12:27.475320 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:27.475285 4409 generic.go:356] "Generic (PLEG): container finished" podID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerID="094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725" exitCode=0 Nov 03 07:12:27.475888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:27.475365 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" event={"ID":"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66","Type":"ContainerDied","Data":"094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725"} Nov 03 07:12:27.475968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:27.475911 4409 scope.go:122] "RemoveContainer" containerID="094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725" Nov 03 07:12:27.477252 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:27.477225 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:12:27.477352 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:27.477304 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:12:27.477419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:27.477348 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:12:28.495473 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:28.495434 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" event={"ID":"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66","Type":"ContainerStarted","Data":"ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f"} Nov 03 07:12:29.147695 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:29.147657 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:12:29.147911 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:29.147889 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 07:12:29.147965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:29.147945 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 07:12:30.147923 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:30.147894 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 07:12:30.148330 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:30.147951 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 07:12:31.147688 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:31.147656 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 07:12:31.147876 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:31.147726 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 07:12:32.147814 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:32.147782 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 07:12:32.148206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:32.147832 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 07:12:33.147665 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:33.147626 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 07:12:33.147893 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:33.147690 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 07:12:34.152043 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:34.152017 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:12:34.152519 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:34.152408 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:12:34.152998 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:12:34.152983 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:12:38.776745 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:12:38.776694 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:13:08.773855 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:13:08.773816 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:13:38.774853 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:13:38.774815 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:14:04.981189 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:04.981163 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:14:04.983207 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:04.983185 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:14:08.774303 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:14:08.774269 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:14:28.068527 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.068489 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw"] Nov 03 07:14:28.072105 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.072084 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.150810 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.150777 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tls-certs\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.151017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.150824 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-dshm\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.151017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.150915 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-home\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.151139 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.151016 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-model-cache\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.151139 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.151081 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8j2w\" (UniqueName: \"kubernetes.io/projected/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-kube-api-access-x8j2w\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.151247 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.151146 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tmp-dir\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.175002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.174968 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"default-dockercfg-tkq4z\"" Nov 03 07:14:28.175002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.174978 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"openshift-service-ca.crt\"" Nov 03 07:14:28.175256 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.175064 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"premium-w3-1f2e7e-kserve-self-signed-certs\"" Nov 03 07:14:28.175256 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.175137 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"kube-root-ca.crt\"" Nov 03 07:14:28.177860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.177835 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw"] Nov 03 07:14:28.252850 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.252810 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-home\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.253035 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.252893 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-model-cache\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.253035 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.252921 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8j2w\" (UniqueName: \"kubernetes.io/projected/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-kube-api-access-x8j2w\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.253035 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.252973 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tmp-dir\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.253170 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.253049 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tls-certs\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.253242 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.253221 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-home\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.253285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.253228 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-dshm\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.253329 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.253290 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-model-cache\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.253380 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.253363 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tmp-dir\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.255411 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.255393 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-dshm\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.255761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.255745 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tls-certs\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.316350 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.316317 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8j2w\" (UniqueName: \"kubernetes.io/projected/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-kube-api-access-x8j2w\") pod \"premium-w3-1f2e7e-kserve-b658b5565-zl8cw\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.384273 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.384188 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:28.790597 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:28.790569 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw"] Nov 03 07:14:28.792525 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:14:28.792494 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78d7765c_f51f_43d5_9cbc_9cc1dd5dc36c.slice/crio-d3fcad00e31d55ea41bd8f63658b4c893b91843e4b41ec8c09ab8153853ffc50 WatchSource:0}: Error finding container d3fcad00e31d55ea41bd8f63658b4c893b91843e4b41ec8c09ab8153853ffc50: Status 404 returned error can't find the container with id d3fcad00e31d55ea41bd8f63658b4c893b91843e4b41ec8c09ab8153853ffc50 Nov 03 07:14:29.115822 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:29.115783 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" event={"ID":"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c","Type":"ContainerStarted","Data":"c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e"} Nov 03 07:14:29.116260 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:29.115833 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" event={"ID":"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c","Type":"ContainerStarted","Data":"d3fcad00e31d55ea41bd8f63658b4c893b91843e4b41ec8c09ab8153853ffc50"} Nov 03 07:14:29.235551 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:29.235507 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podStartSLOduration=1.235492888 podStartE2EDuration="1.235492888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:14:29.233693635 +0000 UTC m=+924.426235386" watchObservedRunningTime="2026-11-03 07:14:29.235492888 +0000 UTC m=+924.428034638" Nov 03 07:14:29.385280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:29.385191 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:38.384625 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:38.384584 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:38.397319 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:38.397294 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:38.774864 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:14:38.774768 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:14:39.189015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:39.188987 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:14:51.910761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:51.910678 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm"] Nov 03 07:14:51.914495 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:51.914479 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.013696 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.013655 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tmp-dir\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.013884 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.013774 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9w6l\" (UniqueName: \"kubernetes.io/projected/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-kube-api-access-s9w6l\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.014027 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.014004 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-model-cache\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.014091 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.014050 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tls-certs\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.014183 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.014163 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-home\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.014233 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.014217 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-dshm\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.014302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.014284 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-3b41b9\"/\"openshift-service-ca.crt\"" Nov 03 07:14:52.014346 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.014283 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-3b41b9\"/\"kube-root-ca.crt\"" Nov 03 07:14:52.021746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.017602 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-3b41b9\"/\"premium-w6-3b41b9-kserve-self-signed-certs\"" Nov 03 07:14:52.021746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.018795 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-3b41b9\"/\"default-dockercfg-62d8x\"" Nov 03 07:14:52.021746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.020814 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm"] Nov 03 07:14:52.114931 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.114905 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tmp-dir\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.115124 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.114965 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9w6l\" (UniqueName: \"kubernetes.io/projected/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-kube-api-access-s9w6l\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.115124 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.115015 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-model-cache\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.115124 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.115035 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tls-certs\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.115310 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.115149 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-home\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.115310 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.115205 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-dshm\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.115416 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.115397 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tmp-dir\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.115460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.115433 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-model-cache\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.115496 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.115467 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-home\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.117447 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.117420 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-dshm\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.117555 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.117495 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tls-certs\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.180271 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.180207 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9w6l\" (UniqueName: \"kubernetes.io/projected/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-kube-api-access-s9w6l\") pod \"premium-w6-3b41b9-kserve-66cb8b5689-j2pwm\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.226973 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.226942 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:14:52.633969 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:52.633945 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm"] Nov 03 07:14:52.635886 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:14:52.635858 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f2bd54c_5ad9_4521_8c53_71cbe7b2f768.slice/crio-2eeae48c81a91245f6dad17412ad6be7a13d04979154d7447ac89f77489e8654 WatchSource:0}: Error finding container 2eeae48c81a91245f6dad17412ad6be7a13d04979154d7447ac89f77489e8654: Status 404 returned error can't find the container with id 2eeae48c81a91245f6dad17412ad6be7a13d04979154d7447ac89f77489e8654 Nov 03 07:14:53.266374 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:53.266336 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" event={"ID":"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768","Type":"ContainerStarted","Data":"07cd6a070bf48bd270ebd77f58af7c2cb1706e63b89be5803163a8d8825ff86d"} Nov 03 07:14:53.266776 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:53.266385 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" event={"ID":"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768","Type":"ContainerStarted","Data":"2eeae48c81a91245f6dad17412ad6be7a13d04979154d7447ac89f77489e8654"} Nov 03 07:14:53.386839 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:53.386789 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podStartSLOduration=2.386772402 podStartE2EDuration="2.386772402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:14:53.385790511 +0000 UTC m=+948.578332260" watchObservedRunningTime="2026-11-03 07:14:53.386772402 +0000 UTC m=+948.579314151" Nov 03 07:14:54.227731 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:14:54.227682 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:15:00.180834 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.180799 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9"] Nov 03 07:15:00.184952 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.184932 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" Nov 03 07:15:00.201030 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.201004 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrpn5\" (UniqueName: \"kubernetes.io/projected/483feb6d-da9f-4d3c-8023-ded0dadd184c-kube-api-access-hrpn5\") pod \"maas-api-key-cleanup-29894835-qbsr9\" (UID: \"483feb6d-da9f-4d3c-8023-ded0dadd184c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" Nov 03 07:15:00.201138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.201053 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483feb6d-da9f-4d3c-8023-ded0dadd184c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894835-qbsr9\" (UID: \"483feb6d-da9f-4d3c-8023-ded0dadd184c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" Nov 03 07:15:00.206030 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.206005 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j"] Nov 03 07:15:00.210430 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.210410 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" Nov 03 07:15:00.240021 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.239995 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\"" Nov 03 07:15:00.283981 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.283954 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9"] Nov 03 07:15:00.301839 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.301813 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04d81d8-4d11-42be-9b18-27089974b284-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j\" (UID: \"f04d81d8-4d11-42be-9b18-27089974b284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" Nov 03 07:15:00.301987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.301859 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5qv\" (UniqueName: \"kubernetes.io/projected/f04d81d8-4d11-42be-9b18-27089974b284-kube-api-access-vz5qv\") pod \"maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j\" (UID: \"f04d81d8-4d11-42be-9b18-27089974b284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" Nov 03 07:15:00.302060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.302012 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrpn5\" (UniqueName: \"kubernetes.io/projected/483feb6d-da9f-4d3c-8023-ded0dadd184c-kube-api-access-hrpn5\") pod \"maas-api-key-cleanup-29894835-qbsr9\" (UID: \"483feb6d-da9f-4d3c-8023-ded0dadd184c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" Nov 03 07:15:00.302119 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.302060 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483feb6d-da9f-4d3c-8023-ded0dadd184c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894835-qbsr9\" (UID: \"483feb6d-da9f-4d3c-8023-ded0dadd184c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" Nov 03 07:15:00.302727 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.302687 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483feb6d-da9f-4d3c-8023-ded0dadd184c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894835-qbsr9\" (UID: \"483feb6d-da9f-4d3c-8023-ded0dadd184c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" Nov 03 07:15:00.333562 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.333533 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j"] Nov 03 07:15:00.364764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.364739 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrpn5\" (UniqueName: \"kubernetes.io/projected/483feb6d-da9f-4d3c-8023-ded0dadd184c-kube-api-access-hrpn5\") pod \"maas-api-key-cleanup-29894835-qbsr9\" (UID: \"483feb6d-da9f-4d3c-8023-ded0dadd184c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" Nov 03 07:15:00.403515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.403495 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04d81d8-4d11-42be-9b18-27089974b284-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j\" (UID: \"f04d81d8-4d11-42be-9b18-27089974b284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" Nov 03 07:15:00.403628 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.403537 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5qv\" (UniqueName: \"kubernetes.io/projected/f04d81d8-4d11-42be-9b18-27089974b284-kube-api-access-vz5qv\") pod \"maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j\" (UID: \"f04d81d8-4d11-42be-9b18-27089974b284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" Nov 03 07:15:00.404105 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.404086 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04d81d8-4d11-42be-9b18-27089974b284-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j\" (UID: \"f04d81d8-4d11-42be-9b18-27089974b284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" Nov 03 07:15:00.465834 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.465779 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5qv\" (UniqueName: \"kubernetes.io/projected/f04d81d8-4d11-42be-9b18-27089974b284-kube-api-access-vz5qv\") pod \"maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j\" (UID: \"f04d81d8-4d11-42be-9b18-27089974b284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" Nov 03 07:15:00.498751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.498729 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" Nov 03 07:15:00.524199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.524179 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" Nov 03 07:15:00.879268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.879245 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9"] Nov 03 07:15:00.880834 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:15:00.880810 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483feb6d_da9f_4d3c_8023_ded0dadd184c.slice/crio-00399531d2fa6107b83c51250552258b5525a1cc41a5c2bba9125a43f64a3214 WatchSource:0}: Error finding container 00399531d2fa6107b83c51250552258b5525a1cc41a5c2bba9125a43f64a3214: Status 404 returned error can't find the container with id 00399531d2fa6107b83c51250552258b5525a1cc41a5c2bba9125a43f64a3214 Nov 03 07:15:00.882683 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.882667 4409 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:15:00.929651 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:00.929627 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j"] Nov 03 07:15:00.930910 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:15:00.930886 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04d81d8_4d11_42be_9b18_27089974b284.slice/crio-e98229c23e0593d5f6706b7d64149713c7f24778f112b3c006d335559a741429 WatchSource:0}: Error finding container e98229c23e0593d5f6706b7d64149713c7f24778f112b3c006d335559a741429: Status 404 returned error can't find the container with id e98229c23e0593d5f6706b7d64149713c7f24778f112b3c006d335559a741429 Nov 03 07:15:01.312160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:01.312121 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" event={"ID":"483feb6d-da9f-4d3c-8023-ded0dadd184c","Type":"ContainerStarted","Data":"00399531d2fa6107b83c51250552258b5525a1cc41a5c2bba9125a43f64a3214"} Nov 03 07:15:01.313501 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:01.313472 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" event={"ID":"f04d81d8-4d11-42be-9b18-27089974b284","Type":"ContainerStarted","Data":"e98229c23e0593d5f6706b7d64149713c7f24778f112b3c006d335559a741429"} Nov 03 07:15:02.231000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:02.230960 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:15:02.244169 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:02.244143 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:15:02.330120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:02.330089 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:15:05.343696 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:05.343616 4409 generic.go:356] "Generic (PLEG): container finished" podID="f04d81d8-4d11-42be-9b18-27089974b284" containerID="54418ce27a4135d8c12ed1e620bb8301c8acfb4dd7742deabba686bb1c2b9e4d" exitCode=60 Nov 03 07:15:05.344101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:05.343705 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" event={"ID":"f04d81d8-4d11-42be-9b18-27089974b284","Type":"ContainerDied","Data":"54418ce27a4135d8c12ed1e620bb8301c8acfb4dd7742deabba686bb1c2b9e4d"} Nov 03 07:15:05.344101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:05.343938 4409 scope.go:122] "RemoveContainer" containerID="54418ce27a4135d8c12ed1e620bb8301c8acfb4dd7742deabba686bb1c2b9e4d" Nov 03 07:15:05.345481 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:05.345460 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" event={"ID":"483feb6d-da9f-4d3c-8023-ded0dadd184c","Type":"ContainerStarted","Data":"4c08b5548ef3dd21949c9c017cee0de7a8682bb7ad8d0d282bc71a0146c23261"} Nov 03 07:15:05.579615 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:05.579560 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" podStartSLOduration=1.551898615 podStartE2EDuration="5.579543662s" totalImagesPullingTime="4.027645047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:05.579306771 +0000 UTC m=+960.771848525" watchObservedRunningTime="2026-11-03 07:15:05.579543662 +0000 UTC m=+960.772085433" Nov 03 07:15:06.352339 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:06.352311 4409 generic.go:356] "Generic (PLEG): container finished" podID="483feb6d-da9f-4d3c-8023-ded0dadd184c" containerID="4c08b5548ef3dd21949c9c017cee0de7a8682bb7ad8d0d282bc71a0146c23261" exitCode=60 Nov 03 07:15:06.352695 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:06.352355 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" event={"ID":"483feb6d-da9f-4d3c-8023-ded0dadd184c","Type":"ContainerDied","Data":"4c08b5548ef3dd21949c9c017cee0de7a8682bb7ad8d0d282bc71a0146c23261"} Nov 03 07:15:06.352695 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:06.352659 4409 scope.go:122] "RemoveContainer" containerID="4c08b5548ef3dd21949c9c017cee0de7a8682bb7ad8d0d282bc71a0146c23261" Nov 03 07:15:06.354577 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:06.354559 4409 generic.go:356] "Generic (PLEG): container finished" podID="f04d81d8-4d11-42be-9b18-27089974b284" containerID="0d40b944f5c31655eefdad89fd278ec5e2717c72e57af7b2dcb46b40c8c5654c" exitCode=60 Nov 03 07:15:06.354671 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:06.354644 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" event={"ID":"f04d81d8-4d11-42be-9b18-27089974b284","Type":"ContainerDied","Data":"0d40b944f5c31655eefdad89fd278ec5e2717c72e57af7b2dcb46b40c8c5654c"} Nov 03 07:15:06.354767 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:06.354700 4409 scope.go:122] "RemoveContainer" containerID="54418ce27a4135d8c12ed1e620bb8301c8acfb4dd7742deabba686bb1c2b9e4d" Nov 03 07:15:06.354921 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:06.354902 4409 scope.go:122] "RemoveContainer" containerID="0d40b944f5c31655eefdad89fd278ec5e2717c72e57af7b2dcb46b40c8c5654c" Nov 03 07:15:06.355362 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:15:06.355337 4409 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j_odh-ai-gateway-infra(f04d81d8-4d11-42be-9b18-27089974b284)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" podUID="f04d81d8-4d11-42be-9b18-27089974b284" Nov 03 07:15:07.362013 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:07.361981 4409 generic.go:356] "Generic (PLEG): container finished" podID="483feb6d-da9f-4d3c-8023-ded0dadd184c" containerID="07bd39e2831852fa18ecfd6df87f2a65ffa63aa2dfbab186eedb379cb34028ed" exitCode=60 Nov 03 07:15:07.362449 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:07.362060 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" event={"ID":"483feb6d-da9f-4d3c-8023-ded0dadd184c","Type":"ContainerDied","Data":"07bd39e2831852fa18ecfd6df87f2a65ffa63aa2dfbab186eedb379cb34028ed"} Nov 03 07:15:07.362449 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:07.362117 4409 scope.go:122] "RemoveContainer" containerID="4c08b5548ef3dd21949c9c017cee0de7a8682bb7ad8d0d282bc71a0146c23261" Nov 03 07:15:07.362449 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:07.362337 4409 scope.go:122] "RemoveContainer" containerID="07bd39e2831852fa18ecfd6df87f2a65ffa63aa2dfbab186eedb379cb34028ed" Nov 03 07:15:07.362619 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:15:07.362595 4409 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894835-qbsr9_odh-ai-gateway-infra(483feb6d-da9f-4d3c-8023-ded0dadd184c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" Nov 03 07:15:07.364264 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:07.364213 4409 scope.go:122] "RemoveContainer" containerID="0d40b944f5c31655eefdad89fd278ec5e2717c72e57af7b2dcb46b40c8c5654c" Nov 03 07:15:07.364444 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:15:07.364425 4409 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j_odh-ai-gateway-infra(f04d81d8-4d11-42be-9b18-27089974b284)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" podUID="f04d81d8-4d11-42be-9b18-27089974b284" Nov 03 07:15:08.377098 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:08.377053 4409 scope.go:122] "RemoveContainer" containerID="07bd39e2831852fa18ecfd6df87f2a65ffa63aa2dfbab186eedb379cb34028ed" Nov 03 07:15:08.377608 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:15:08.377346 4409 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894835-qbsr9_odh-ai-gateway-infra(483feb6d-da9f-4d3c-8023-ded0dadd184c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" Nov 03 07:15:08.776107 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:15:08.776003 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:15:16.905729 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:16.905693 4409 scope.go:122] "RemoveContainer" containerID="0d40b944f5c31655eefdad89fd278ec5e2717c72e57af7b2dcb46b40c8c5654c" Nov 03 07:15:17.441369 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:17.441284 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" event={"ID":"f04d81d8-4d11-42be-9b18-27089974b284","Type":"ContainerStarted","Data":"f1c8c297276e3334de5e85fc7e9cecb1cd0f67dc02622cc32122ee6493defc1f"} Nov 03 07:15:17.561638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:17.561585 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" podStartSLOduration=14.079223361 podStartE2EDuration="17.561571655s" totalImagesPullingTime="3.482348294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:17.560481046 +0000 UTC m=+972.753022829" watchObservedRunningTime="2026-11-03 07:15:17.561571655 +0000 UTC m=+972.754113411" Nov 03 07:15:18.448849 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:18.448818 4409 generic.go:356] "Generic (PLEG): container finished" podID="f04d81d8-4d11-42be-9b18-27089974b284" containerID="f1c8c297276e3334de5e85fc7e9cecb1cd0f67dc02622cc32122ee6493defc1f" exitCode=60 Nov 03 07:15:18.449263 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:18.448898 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" event={"ID":"f04d81d8-4d11-42be-9b18-27089974b284","Type":"ContainerDied","Data":"f1c8c297276e3334de5e85fc7e9cecb1cd0f67dc02622cc32122ee6493defc1f"} Nov 03 07:15:18.449263 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:18.448962 4409 scope.go:122] "RemoveContainer" containerID="0d40b944f5c31655eefdad89fd278ec5e2717c72e57af7b2dcb46b40c8c5654c" Nov 03 07:15:18.449390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:18.449262 4409 scope.go:122] "RemoveContainer" containerID="f1c8c297276e3334de5e85fc7e9cecb1cd0f67dc02622cc32122ee6493defc1f" Nov 03 07:15:18.449527 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:15:18.449506 4409 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j_odh-ai-gateway-infra(f04d81d8-4d11-42be-9b18-27089974b284)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" podUID="f04d81d8-4d11-42be-9b18-27089974b284" Nov 03 07:15:18.579759 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:18.579736 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j"] Nov 03 07:15:19.591186 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:19.591164 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" Nov 03 07:15:19.710244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:19.710172 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04d81d8-4d11-42be-9b18-27089974b284-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04d81d8-4d11-42be-9b18-27089974b284-maas-api-service-ca\") pod \"f04d81d8-4d11-42be-9b18-27089974b284\" (UID: \"f04d81d8-4d11-42be-9b18-27089974b284\") " Nov 03 07:15:19.710561 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:19.710541 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04d81d8-4d11-42be-9b18-27089974b284-kube-api-access-vz5qv\" (UniqueName: \"kubernetes.io/projected/f04d81d8-4d11-42be-9b18-27089974b284-kube-api-access-vz5qv\") pod \"f04d81d8-4d11-42be-9b18-27089974b284\" (UID: \"f04d81d8-4d11-42be-9b18-27089974b284\") " Nov 03 07:15:19.711101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:19.711074 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04d81d8-4d11-42be-9b18-27089974b284-maas-api-service-ca" pod "f04d81d8-4d11-42be-9b18-27089974b284" (UID: "f04d81d8-4d11-42be-9b18-27089974b284"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:15:19.713113 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:19.713087 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04d81d8-4d11-42be-9b18-27089974b284-kube-api-access-vz5qv" pod "f04d81d8-4d11-42be-9b18-27089974b284" (UID: "f04d81d8-4d11-42be-9b18-27089974b284"). InnerVolumeSpecName "kube-api-access-vz5qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:15:19.812289 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:19.812258 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5qv\" (UniqueName: \"kubernetes.io/projected/f04d81d8-4d11-42be-9b18-27089974b284-kube-api-access-vz5qv\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:15:19.812289 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:19.812285 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04d81d8-4d11-42be-9b18-27089974b284-maas-api-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:15:20.463289 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:20.463259 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" Nov 03 07:15:20.463479 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:20.463300 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j" event={"ID":"f04d81d8-4d11-42be-9b18-27089974b284","Type":"ContainerDied","Data":"e98229c23e0593d5f6706b7d64149713c7f24778f112b3c006d335559a741429"} Nov 03 07:15:20.463479 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:20.463367 4409 scope.go:122] "RemoveContainer" containerID="f1c8c297276e3334de5e85fc7e9cecb1cd0f67dc02622cc32122ee6493defc1f" Nov 03 07:15:20.630030 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:20.630003 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j"] Nov 03 07:15:20.641664 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:20.641642 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-3b41b9-29894835-vzk2j"] Nov 03 07:15:20.906329 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:20.906302 4409 scope.go:122] "RemoveContainer" containerID="07bd39e2831852fa18ecfd6df87f2a65ffa63aa2dfbab186eedb379cb34028ed" Nov 03 07:15:20.913091 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:20.913069 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04d81d8-4d11-42be-9b18-27089974b284" path="/var/lib/kubelet/pods/f04d81d8-4d11-42be-9b18-27089974b284/volumes" Nov 03 07:15:21.471621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:21.471546 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" event={"ID":"483feb6d-da9f-4d3c-8023-ded0dadd184c","Type":"ContainerStarted","Data":"07c5b153f1a08c335ff69fa12b389e91c8c0580ff09c8091f96cc609aeb5525f"} Nov 03 07:15:22.478864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:22.478829 4409 generic.go:356] "Generic (PLEG): container finished" podID="483feb6d-da9f-4d3c-8023-ded0dadd184c" containerID="07c5b153f1a08c335ff69fa12b389e91c8c0580ff09c8091f96cc609aeb5525f" exitCode=60 Nov 03 07:15:22.478864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:22.478864 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" event={"ID":"483feb6d-da9f-4d3c-8023-ded0dadd184c","Type":"ContainerDied","Data":"07c5b153f1a08c335ff69fa12b389e91c8c0580ff09c8091f96cc609aeb5525f"} Nov 03 07:15:22.479298 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:22.478911 4409 scope.go:122] "RemoveContainer" containerID="07bd39e2831852fa18ecfd6df87f2a65ffa63aa2dfbab186eedb379cb34028ed" Nov 03 07:15:22.479298 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:22.479171 4409 scope.go:122] "RemoveContainer" containerID="07c5b153f1a08c335ff69fa12b389e91c8c0580ff09c8091f96cc609aeb5525f" Nov 03 07:15:22.479435 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:15:22.479412 4409 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894835-qbsr9_odh-ai-gateway-infra(483feb6d-da9f-4d3c-8023-ded0dadd184c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" Nov 03 07:15:22.610330 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:22.610307 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9"] Nov 03 07:15:23.626482 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:23.626458 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" Nov 03 07:15:23.757352 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:23.757275 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/483feb6d-da9f-4d3c-8023-ded0dadd184c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483feb6d-da9f-4d3c-8023-ded0dadd184c-maas-api-service-ca\") pod \"483feb6d-da9f-4d3c-8023-ded0dadd184c\" (UID: \"483feb6d-da9f-4d3c-8023-ded0dadd184c\") " Nov 03 07:15:23.757352 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:23.757314 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/483feb6d-da9f-4d3c-8023-ded0dadd184c-kube-api-access-hrpn5\" (UniqueName: \"kubernetes.io/projected/483feb6d-da9f-4d3c-8023-ded0dadd184c-kube-api-access-hrpn5\") pod \"483feb6d-da9f-4d3c-8023-ded0dadd184c\" (UID: \"483feb6d-da9f-4d3c-8023-ded0dadd184c\") " Nov 03 07:15:23.757675 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:23.757647 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483feb6d-da9f-4d3c-8023-ded0dadd184c-maas-api-service-ca" pod "483feb6d-da9f-4d3c-8023-ded0dadd184c" (UID: "483feb6d-da9f-4d3c-8023-ded0dadd184c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:15:23.759628 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:23.759600 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483feb6d-da9f-4d3c-8023-ded0dadd184c-kube-api-access-hrpn5" pod "483feb6d-da9f-4d3c-8023-ded0dadd184c" (UID: "483feb6d-da9f-4d3c-8023-ded0dadd184c"). InnerVolumeSpecName "kube-api-access-hrpn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:15:23.858464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:23.858432 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483feb6d-da9f-4d3c-8023-ded0dadd184c-maas-api-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:15:23.858464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:23.858460 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrpn5\" (UniqueName: \"kubernetes.io/projected/483feb6d-da9f-4d3c-8023-ded0dadd184c-kube-api-access-hrpn5\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:15:24.492862 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:24.492833 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" Nov 03 07:15:24.493061 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:24.492878 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9" event={"ID":"483feb6d-da9f-4d3c-8023-ded0dadd184c","Type":"ContainerDied","Data":"00399531d2fa6107b83c51250552258b5525a1cc41a5c2bba9125a43f64a3214"} Nov 03 07:15:24.493061 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:24.492944 4409 scope.go:122] "RemoveContainer" containerID="07c5b153f1a08c335ff69fa12b389e91c8c0580ff09c8091f96cc609aeb5525f" Nov 03 07:15:24.663345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:24.663314 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9"] Nov 03 07:15:24.713072 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:24.713046 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894835-qbsr9"] Nov 03 07:15:24.912817 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:24.912784 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" path="/var/lib/kubelet/pods/483feb6d-da9f-4d3c-8023-ded0dadd184c/volumes" Nov 03 07:15:38.773131 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:15:38.773087 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:15:52.055117 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.055079 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n"] Nov 03 07:15:52.055810 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.055792 4409 cpu_manager.go:414] "containerMap: removing container" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" containerName="cleanup" Nov 03 07:15:52.055917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.055812 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" containerName="cleanup" Nov 03 07:15:52.055917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.055837 4409 cpu_manager.go:414] "containerMap: removing container" podUID="f04d81d8-4d11-42be-9b18-27089974b284" containerName="cleanup" Nov 03 07:15:52.055917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.055844 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04d81d8-4d11-42be-9b18-27089974b284" containerName="cleanup" Nov 03 07:15:52.055917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.055870 4409 cpu_manager.go:414] "containerMap: removing container" podUID="f04d81d8-4d11-42be-9b18-27089974b284" containerName="cleanup" Nov 03 07:15:52.055917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.055877 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04d81d8-4d11-42be-9b18-27089974b284" containerName="cleanup" Nov 03 07:15:52.055917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.055910 4409 cpu_manager.go:414] "containerMap: removing container" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" containerName="cleanup" Nov 03 07:15:52.055917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.055917 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" containerName="cleanup" Nov 03 07:15:52.056246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.056041 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04d81d8-4d11-42be-9b18-27089974b284" containerName="cleanup" Nov 03 07:15:52.056246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.056054 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04d81d8-4d11-42be-9b18-27089974b284" containerName="cleanup" Nov 03 07:15:52.056246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.056065 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" containerName="cleanup" Nov 03 07:15:52.056246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.056075 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" containerName="cleanup" Nov 03 07:15:52.059348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.059327 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.113915 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.113896 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1f2e7e-openshift-default-dockercfg-plh6k\"" Nov 03 07:15:52.164693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.164665 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n"] Nov 03 07:15:52.218545 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.218512 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.218545 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.218544 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jltx8\" (UniqueName: \"kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-kube-api-access-jltx8\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.218749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.218570 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.218749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.218633 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.218838 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.218753 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.218838 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.218800 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.218913 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.218866 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459e890f-044a-4949-a9b8-6a1f40394cd0-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.218913 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.218899 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.219141 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.218933 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.320236 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320160 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.320236 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320198 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.320466 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320251 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.320466 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320307 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.320466 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320359 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459e890f-044a-4949-a9b8-6a1f40394cd0-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.320466 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320419 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.320683 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320473 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.320683 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320508 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.320683 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320537 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jltx8\" (UniqueName: \"kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-kube-api-access-jltx8\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.320683 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320661 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.320925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320692 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.320925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320906 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.321028 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.320955 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.321140 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.321117 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459e890f-044a-4949-a9b8-6a1f40394cd0-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.323046 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.323028 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.323279 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.323257 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.397779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.384656 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.398833 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.398768 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jltx8\" (UniqueName: \"kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-kube-api-access-jltx8\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:52.677355 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:52.676995 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:53.211260 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:53.211231 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n"] Nov 03 07:15:53.218799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:53.218764 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:15:53.218986 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:53.218968 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:15:53.219130 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:53.219116 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:15:53.664060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:53.664021 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" event={"ID":"459e890f-044a-4949-a9b8-6a1f40394cd0","Type":"ContainerStarted","Data":"cfb8c74287fc843902759272633fb6f3e5bc1817c2eaca7aa9fb0c9a00a683bf"} Nov 03 07:15:53.664258 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:53.664088 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" event={"ID":"459e890f-044a-4949-a9b8-6a1f40394cd0","Type":"ContainerStarted","Data":"6640bca635f208622f11796f8cc57df9f276ab6236030621565ac51ac3c8a735"} Nov 03 07:15:53.678864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:53.678798 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:15:53.830587 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:53.830324 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" podStartSLOduration=2.830302744 podStartE2EDuration="2.830302744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:15:53.827481911 +0000 UTC m=+1009.020023653" watchObservedRunningTime="2026-11-03 07:15:53.830302744 +0000 UTC m=+1009.022844496" Nov 03 07:15:54.692872 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:54.692836 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 07:15:54.693478 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:54.693441 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" podUID="459e890f-044a-4949-a9b8-6a1f40394cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 07:15:55.686117 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:55.686078 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 07:15:55.686373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:55.686348 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" podUID="459e890f-044a-4949-a9b8-6a1f40394cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 07:15:56.683786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:56.683750 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 07:15:56.684390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:56.684357 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" podUID="459e890f-044a-4949-a9b8-6a1f40394cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 07:15:57.684735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:57.680043 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 07:15:57.684735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:57.680101 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" podUID="459e890f-044a-4949-a9b8-6a1f40394cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 07:15:58.696607 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:58.696125 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 07:15:58.696607 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:58.696186 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" podUID="459e890f-044a-4949-a9b8-6a1f40394cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 07:15:59.678075 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:59.678034 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 07:15:59.678234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:59.678103 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" podUID="459e890f-044a-4949-a9b8-6a1f40394cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 07:15:59.741905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:59.741872 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm"] Nov 03 07:15:59.742730 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:15:59.742679 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" podUID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" containerName="istio-proxy" containerID="cri-o://428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c" gracePeriod=30 Nov 03 07:16:00.682767 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:00.682739 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:16:00.683095 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:00.683076 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:16:00.683673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:00.683654 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:16:04.995429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:04.995406 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:16:05.103331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103304 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istiod-ca-cert\") pod \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " Nov 03 07:16:05.103504 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103366 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-credential-socket\") pod \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " Nov 03 07:16:05.103504 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103448 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-envoy\") pod \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " Nov 03 07:16:05.103504 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103478 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-socket\") pod \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " Nov 03 07:16:05.103667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103513 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-token\" (UniqueName: \"kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-token\") pod \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " Nov 03 07:16:05.103667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103541 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-kube-api-access-pfh44\" (UniqueName: \"kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-kube-api-access-pfh44\") pod \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " Nov 03 07:16:05.103667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103569 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-credential-socket" pod "68f4d0e8-3e95-44ca-a5d3-3109d2120453" (UID: "68f4d0e8-3e95-44ca-a5d3-3109d2120453"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:16:05.103667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103581 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-podinfo\") pod \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " Nov 03 07:16:05.103667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103640 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-data\") pod \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " Nov 03 07:16:05.103956 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103742 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-socket" pod "68f4d0e8-3e95-44ca-a5d3-3109d2120453" (UID: "68f4d0e8-3e95-44ca-a5d3-3109d2120453"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:16:05.103956 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103752 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-certs\") pod \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\" (UID: \"68f4d0e8-3e95-44ca-a5d3-3109d2120453\") " Nov 03 07:16:05.103956 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103783 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istiod-ca-cert" pod "68f4d0e8-3e95-44ca-a5d3-3109d2120453" (UID: "68f4d0e8-3e95-44ca-a5d3-3109d2120453"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:16:05.103956 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.103950 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-data" pod "68f4d0e8-3e95-44ca-a5d3-3109d2120453" (UID: "68f4d0e8-3e95-44ca-a5d3-3109d2120453"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:16:05.104177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.104071 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-certs" pod "68f4d0e8-3e95-44ca-a5d3-3109d2120453" (UID: "68f4d0e8-3e95-44ca-a5d3-3109d2120453"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:16:05.104256 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.104240 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:16:05.104486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.104257 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:16:05.104486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.104268 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:16:05.104486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.104276 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:16:05.104486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.104284 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:16:05.105753 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.105726 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-envoy" pod "68f4d0e8-3e95-44ca-a5d3-3109d2120453" (UID: "68f4d0e8-3e95-44ca-a5d3-3109d2120453"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:16:05.105907 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.105882 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-token" pod "68f4d0e8-3e95-44ca-a5d3-3109d2120453" (UID: "68f4d0e8-3e95-44ca-a5d3-3109d2120453"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:16:05.105907 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.105890 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-podinfo" pod "68f4d0e8-3e95-44ca-a5d3-3109d2120453" (UID: "68f4d0e8-3e95-44ca-a5d3-3109d2120453"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:16:05.106189 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.106169 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-kube-api-access-pfh44" pod "68f4d0e8-3e95-44ca-a5d3-3109d2120453" (UID: "68f4d0e8-3e95-44ca-a5d3-3109d2120453"). InnerVolumeSpecName "kube-api-access-pfh44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:16:05.205609 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.205583 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:16:05.205609 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.205606 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:16:05.205819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.205616 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfh44\" (UniqueName: \"kubernetes.io/projected/68f4d0e8-3e95-44ca-a5d3-3109d2120453-kube-api-access-pfh44\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:16:05.205819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.205626 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f4d0e8-3e95-44ca-a5d3-3109d2120453-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:16:05.778937 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.778902 4409 generic.go:356] "Generic (PLEG): container finished" podID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" containerID="428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c" exitCode=0 Nov 03 07:16:05.779136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.778954 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" Nov 03 07:16:05.779136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.778992 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" event={"ID":"68f4d0e8-3e95-44ca-a5d3-3109d2120453","Type":"ContainerDied","Data":"428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c"} Nov 03 07:16:05.779136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.779058 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm" event={"ID":"68f4d0e8-3e95-44ca-a5d3-3109d2120453","Type":"ContainerDied","Data":"c0dbd15eb9f0f8d95de7a5bec39aa952672a3c5b7dca287fe4103aff026fddbd"} Nov 03 07:16:05.779136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.779093 4409 scope.go:122] "RemoveContainer" containerID="428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c" Nov 03 07:16:05.788751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.788736 4409 scope.go:122] "RemoveContainer" containerID="428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c" Nov 03 07:16:05.789000 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:16:05.788982 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c\": container with ID starting with 428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c not found: ID does not exist" containerID="428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c" Nov 03 07:16:05.789066 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.789012 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c"} err="failed to get container status \"428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c\": rpc error: code = NotFound desc = could not find container \"428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c\": container with ID starting with 428795f369d3e0c765ec70e74d15041c6eb508dab6a7ac45e09a46bd651d5e0c not found: ID does not exist" Nov 03 07:16:05.948973 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.948947 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm"] Nov 03 07:16:05.998417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:05.998392 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-x59tm"] Nov 03 07:16:06.912589 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:06.912557 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" path="/var/lib/kubelet/pods/68f4d0e8-3e95-44ca-a5d3-3109d2120453/volumes" Nov 03 07:16:08.774139 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:16:08.774098 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:16:38.774654 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:16:38.774615 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:16:45.116216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.116182 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg"] Nov 03 07:16:45.116938 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.116920 4409 cpu_manager.go:414] "containerMap: removing container" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" containerName="cleanup" Nov 03 07:16:45.117002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.116941 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" containerName="cleanup" Nov 03 07:16:45.117002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.116966 4409 cpu_manager.go:414] "containerMap: removing container" podUID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" containerName="istio-proxy" Nov 03 07:16:45.117002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.116974 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" containerName="istio-proxy" Nov 03 07:16:45.117002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.117000 4409 cpu_manager.go:414] "containerMap: removing container" podUID="f04d81d8-4d11-42be-9b18-27089974b284" containerName="cleanup" Nov 03 07:16:45.117127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.117008 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04d81d8-4d11-42be-9b18-27089974b284" containerName="cleanup" Nov 03 07:16:45.117160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.117135 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="483feb6d-da9f-4d3c-8023-ded0dadd184c" containerName="cleanup" Nov 03 07:16:45.117160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.117149 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f4d0e8-3e95-44ca-a5d3-3109d2120453" containerName="istio-proxy" Nov 03 07:16:45.117221 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.117161 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04d81d8-4d11-42be-9b18-27089974b284" containerName="cleanup" Nov 03 07:16:45.122140 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.122118 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.163002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.162952 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-tmp-dir\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.163171 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.163105 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27976e39-e6a3-4269-966b-67cca0cf2d9e-tls-certs\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.163171 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.163143 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-model-cache\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.163299 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.163279 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-dshm\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.163336 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.163315 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-home\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.163482 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.163458 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hgp9\" (UniqueName: \"kubernetes.io/projected/27976e39-e6a3-4269-966b-67cca0cf2d9e-kube-api-access-5hgp9\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.225960 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.225932 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1f2e7e\"/\"distinct-w3-1f2e7e-kserve-self-signed-certs\"" Nov 03 07:16:45.228481 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.228462 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg"] Nov 03 07:16:45.282227 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.282181 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-dshm\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.282986 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.282960 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-home\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.283591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.283573 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hgp9\" (UniqueName: \"kubernetes.io/projected/27976e39-e6a3-4269-966b-67cca0cf2d9e-kube-api-access-5hgp9\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.288462 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.288440 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-tmp-dir\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.289038 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.289020 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27976e39-e6a3-4269-966b-67cca0cf2d9e-tls-certs\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.289755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.289735 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-model-cache\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.290281 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.288908 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-tmp-dir\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.290429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.290142 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-model-cache\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.290541 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.283468 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-home\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.294622 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.294595 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27976e39-e6a3-4269-966b-67cca0cf2d9e-tls-certs\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.295263 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.295215 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-dshm\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.347383 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.347359 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hgp9\" (UniqueName: \"kubernetes.io/projected/27976e39-e6a3-4269-966b-67cca0cf2d9e-kube-api-access-5hgp9\") pod \"distinct-w3-1f2e7e-kserve-799769bfcb-56twg\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.434225 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.434165 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:45.839312 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:45.839285 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg"] Nov 03 07:16:45.840560 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:16:45.840537 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27976e39_e6a3_4269_966b_67cca0cf2d9e.slice/crio-16fcce7a26a7d74549595be428d393006aa1dc3b0997627442156852ed22133f WatchSource:0}: Error finding container 16fcce7a26a7d74549595be428d393006aa1dc3b0997627442156852ed22133f: Status 404 returned error can't find the container with id 16fcce7a26a7d74549595be428d393006aa1dc3b0997627442156852ed22133f Nov 03 07:16:46.035121 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:46.035085 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" event={"ID":"27976e39-e6a3-4269-966b-67cca0cf2d9e","Type":"ContainerStarted","Data":"74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae"} Nov 03 07:16:46.035297 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:46.035130 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" event={"ID":"27976e39-e6a3-4269-966b-67cca0cf2d9e","Type":"ContainerStarted","Data":"16fcce7a26a7d74549595be428d393006aa1dc3b0997627442156852ed22133f"} Nov 03 07:16:46.154216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:46.154093 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podStartSLOduration=1.154073772 podStartE2EDuration="1.154073772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:16:46.153774317 +0000 UTC m=+1061.346316068" watchObservedRunningTime="2026-11-03 07:16:46.154073772 +0000 UTC m=+1061.346615523" Nov 03 07:16:46.434355 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:46.434283 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:55.435222 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:55.435190 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:55.448205 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:55.448179 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:16:56.106401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:16:56.106371 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:17:08.777529 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:17:08.777482 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:17:15.379990 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:15.379952 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr"] Nov 03 07:17:15.380586 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:15.380152 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" podUID="6e3e173c-8bab-4805-8a40-fd77e4941d22" containerName="istio-proxy" containerID="cri-o://b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0" gracePeriod=30 Nov 03 07:17:20.640322 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.640300 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:17:20.708793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.708739 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e3e173c-8bab-4805-8a40-fd77e4941d22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3e173c-8bab-4805-8a40-fd77e4941d22-istiod-ca-cert\") pod \"6e3e173c-8bab-4805-8a40-fd77e4941d22\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " Nov 03 07:17:20.708904 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.708793 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-credential-socket\") pod \"6e3e173c-8bab-4805-8a40-fd77e4941d22\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " Nov 03 07:17:20.708904 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.708845 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-data\") pod \"6e3e173c-8bab-4805-8a40-fd77e4941d22\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " Nov 03 07:17:20.708904 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.708890 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-kube-api-access-fktdj\" (UniqueName: \"kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-kube-api-access-fktdj\") pod \"6e3e173c-8bab-4805-8a40-fd77e4941d22\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " Nov 03 07:17:20.709071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.708926 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-token\" (UniqueName: \"kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-token\") pod \"6e3e173c-8bab-4805-8a40-fd77e4941d22\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " Nov 03 07:17:20.709071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.708991 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-envoy\") pod \"6e3e173c-8bab-4805-8a40-fd77e4941d22\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " Nov 03 07:17:20.709071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709029 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-podinfo\") pod \"6e3e173c-8bab-4805-8a40-fd77e4941d22\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " Nov 03 07:17:20.709217 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709084 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-socket\") pod \"6e3e173c-8bab-4805-8a40-fd77e4941d22\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " Nov 03 07:17:20.709217 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709111 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-certs\") pod \"6e3e173c-8bab-4805-8a40-fd77e4941d22\" (UID: \"6e3e173c-8bab-4805-8a40-fd77e4941d22\") " Nov 03 07:17:20.709217 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709125 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-credential-socket" pod "6e3e173c-8bab-4805-8a40-fd77e4941d22" (UID: "6e3e173c-8bab-4805-8a40-fd77e4941d22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:17:20.709217 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709195 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-data" pod "6e3e173c-8bab-4805-8a40-fd77e4941d22" (UID: "6e3e173c-8bab-4805-8a40-fd77e4941d22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:17:20.709407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709242 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3e173c-8bab-4805-8a40-fd77e4941d22-istiod-ca-cert" pod "6e3e173c-8bab-4805-8a40-fd77e4941d22" (UID: "6e3e173c-8bab-4805-8a40-fd77e4941d22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:17:20.709500 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709477 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-certs" pod "6e3e173c-8bab-4805-8a40-fd77e4941d22" (UID: "6e3e173c-8bab-4805-8a40-fd77e4941d22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:17:20.709500 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709491 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-socket" pod "6e3e173c-8bab-4805-8a40-fd77e4941d22" (UID: "6e3e173c-8bab-4805-8a40-fd77e4941d22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:17:20.709879 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709611 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:17:20.709879 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709631 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:17:20.709879 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709645 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3e173c-8bab-4805-8a40-fd77e4941d22-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:17:20.709879 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709658 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:17:20.709879 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.709672 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:17:20.711387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.711361 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-kube-api-access-fktdj" pod "6e3e173c-8bab-4805-8a40-fd77e4941d22" (UID: "6e3e173c-8bab-4805-8a40-fd77e4941d22"). InnerVolumeSpecName "kube-api-access-fktdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:17:20.711475 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.711410 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-envoy" pod "6e3e173c-8bab-4805-8a40-fd77e4941d22" (UID: "6e3e173c-8bab-4805-8a40-fd77e4941d22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:17:20.711475 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.711407 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-podinfo" pod "6e3e173c-8bab-4805-8a40-fd77e4941d22" (UID: "6e3e173c-8bab-4805-8a40-fd77e4941d22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:17:20.711546 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.711459 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-token" pod "6e3e173c-8bab-4805-8a40-fd77e4941d22" (UID: "6e3e173c-8bab-4805-8a40-fd77e4941d22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:17:20.811264 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.811236 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:17:20.811264 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.811265 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:17:20.811445 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.811279 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fktdj\" (UniqueName: \"kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-kube-api-access-fktdj\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:17:20.811445 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:20.811292 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3e173c-8bab-4805-8a40-fd77e4941d22-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:17:21.258012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.257978 4409 generic.go:356] "Generic (PLEG): container finished" podID="6e3e173c-8bab-4805-8a40-fd77e4941d22" containerID="b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0" exitCode=0 Nov 03 07:17:21.258198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.258029 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" Nov 03 07:17:21.258198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.258061 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" event={"ID":"6e3e173c-8bab-4805-8a40-fd77e4941d22","Type":"ContainerDied","Data":"b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0"} Nov 03 07:17:21.258198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.258122 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr" event={"ID":"6e3e173c-8bab-4805-8a40-fd77e4941d22","Type":"ContainerDied","Data":"14b444a67a6223e1c44301c930d2778e80f07802d106b065f6bfc2cc90b8b3ea"} Nov 03 07:17:21.258198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.258151 4409 scope.go:122] "RemoveContainer" containerID="b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0" Nov 03 07:17:21.267267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.267250 4409 scope.go:122] "RemoveContainer" containerID="b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0" Nov 03 07:17:21.267543 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:17:21.267526 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0\": container with ID starting with b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0 not found: ID does not exist" containerID="b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0" Nov 03 07:17:21.267595 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.267552 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0"} err="failed to get container status \"b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0\": rpc error: code = NotFound desc = could not find container \"b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0\": container with ID starting with b364c76d5e5e96e090fe85baab31ad27fa8c40fe0a7e716c7063c7f593c18de0 not found: ID does not exist" Nov 03 07:17:21.477167 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.477141 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr"] Nov 03 07:17:21.526462 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.526400 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-dk9tr"] Nov 03 07:17:21.919354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.919326 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m"] Nov 03 07:17:21.920089 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.920068 4409 cpu_manager.go:414] "containerMap: removing container" podUID="6e3e173c-8bab-4805-8a40-fd77e4941d22" containerName="istio-proxy" Nov 03 07:17:21.920089 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.920090 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3e173c-8bab-4805-8a40-fd77e4941d22" containerName="istio-proxy" Nov 03 07:17:21.920304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.920238 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3e173c-8bab-4805-8a40-fd77e4941d22" containerName="istio-proxy" Nov 03 07:17:21.923477 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.923456 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5"] Nov 03 07:17:21.923612 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.923589 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:21.927018 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.927003 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:21.941686 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.941665 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77"] Nov 03 07:17:21.945241 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.945225 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:21.979753 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.979702 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9697f7-openshift-default-dockercfg-pbdkl\"" Nov 03 07:17:21.979864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:21.979826 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-3b41b9-openshift-default-dockercfg-zfqkd\"" Nov 03 07:17:22.024245 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.024218 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-data\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.024445 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.024399 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6xkr\" (UniqueName: \"kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-kube-api-access-j6xkr\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.024535 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.024478 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.024535 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.024525 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.024657 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.024558 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.024730 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.024640 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.025128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.024813 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e54c4bd-d566-42ab-8613-67b296c03c18-istiod-ca-cert\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.025128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.024844 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.025128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.024899 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.025128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.024924 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk52m\" (UniqueName: \"kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-kube-api-access-mk52m\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.025128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.024957 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-certs\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.025128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.024991 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.025128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025014 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.025128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025035 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.025128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025078 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-credential-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.025128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025105 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.025697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025168 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.025697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025214 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.025697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025235 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.025697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025265 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-podinfo\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.025697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025295 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.025697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025330 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-token\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.025697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025358 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-envoy\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.025697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025444 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.025697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025475 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.025697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025541 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb2px\" (UniqueName: \"kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-kube-api-access-vb2px\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.025697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.025572 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.027736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.027532 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt"] Nov 03 07:17:22.041704 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.040002 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m"] Nov 03 07:17:22.041704 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.040135 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.079812 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.079780 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77"] Nov 03 07:17:22.081472 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.081447 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5"] Nov 03 07:17:22.092582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.092554 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt"] Nov 03 07:17:22.126530 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126504 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.126678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126548 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.126678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126582 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-token\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.126678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126606 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdswf\" (UniqueName: \"kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-kube-api-access-bdswf\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.126678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126636 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-envoy\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126679 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c96119-0bed-477e-9992-eb73a9761fca-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126773 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126803 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126826 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c96119-0bed-477e-9992-eb73a9761fca-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126866 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2px\" (UniqueName: \"kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-kube-api-access-vb2px\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126896 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126920 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126969 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-data\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.126993 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6xkr\" (UniqueName: \"kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-kube-api-access-j6xkr\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127028 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127053 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127080 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127106 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127164 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e54c4bd-d566-42ab-8613-67b296c03c18-istiod-ca-cert\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127190 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127225 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.127250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127250 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk52m\" (UniqueName: \"kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-kube-api-access-mk52m\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.128987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127309 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-certs\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.128987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127354 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.128987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127377 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.128987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127402 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.128987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127743 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.128987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.127771 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.128987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.128025 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-data\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.128987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.128257 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.128987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.128559 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.128987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.128850 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e54c4bd-d566-42ab-8613-67b296c03c18-istiod-ca-cert\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.128987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.128897 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.129569 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.129062 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-certs\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.129569 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.129468 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.130084 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.129963 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.130258 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.130226 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.130686 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.130429 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.130686 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.130537 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-credential-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.130686 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.130577 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.130989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.130849 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-credential-socket\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.130989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.130935 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.130989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.130976 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.131239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.131219 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.131360 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.131341 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.131423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.131374 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.131423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.131404 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.131526 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.131426 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.131526 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.131454 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-podinfo\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.131958 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.131933 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.132203 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.132183 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.132326 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.132183 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.132418 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.132398 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.134366 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.134341 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.134571 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.134546 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.135063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.135038 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-envoy\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.135622 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.135598 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-podinfo\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.180015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.179951 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ffee69-openshift-default-dockercfg-mdk9q\"" Nov 03 07:17:22.190383 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.190351 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6xkr\" (UniqueName: \"kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-kube-api-access-j6xkr\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.190795 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.190461 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-token\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.195178 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.195154 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.195559 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.195537 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb2px\" (UniqueName: \"kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-kube-api-access-vb2px\") pod \"e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.195926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.195905 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.196025 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.196008 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk52m\" (UniqueName: \"kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-kube-api-access-mk52m\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.232574 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.232550 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.232674 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.232578 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.232674 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.232596 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.232800 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.232705 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.232844 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.232798 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.232844 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.232831 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdswf\" (UniqueName: \"kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-kube-api-access-bdswf\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.232946 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.232878 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c96119-0bed-477e-9992-eb73a9761fca-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.232946 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.232938 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c96119-0bed-477e-9992-eb73a9761fca-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.233043 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.232944 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.233043 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.232994 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.233043 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.233003 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.233289 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.233253 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.233460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.233396 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.233855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.233827 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c96119-0bed-477e-9992-eb73a9761fca-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.235008 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.234985 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.235289 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.235274 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c96119-0bed-477e-9992-eb73a9761fca-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.238249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.238228 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:22.243695 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.243675 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:22.257512 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.257486 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:22.303511 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.303467 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.303892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.303780 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdswf\" (UniqueName: \"kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-kube-api-access-bdswf\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.369211 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.369184 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:22.632173 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.632145 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m"] Nov 03 07:17:22.633401 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:17:22.633376 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74eb1078_e4b7_41f0_a4a8_5b8af714bf2e.slice/crio-f529d09248dd5e8498b43f4700dfa3406774ee2f8f13e1a0fbf9d0d01f4429e1 WatchSource:0}: Error finding container f529d09248dd5e8498b43f4700dfa3406774ee2f8f13e1a0fbf9d0d01f4429e1: Status 404 returned error can't find the container with id f529d09248dd5e8498b43f4700dfa3406774ee2f8f13e1a0fbf9d0d01f4429e1 Nov 03 07:17:22.636029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.635998 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:17:22.636109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.636066 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:17:22.636109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.636097 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:17:22.681925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.681867 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5"] Nov 03 07:17:22.684196 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.684168 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77"] Nov 03 07:17:22.684836 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:17:22.684814 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e54c4bd_d566_42ab_8613_67b296c03c18.slice/crio-2e78b0b2a61af65ca840fc9cfe5a5b75793a61ce4a908be6b36c2613c3254c5c WatchSource:0}: Error finding container 2e78b0b2a61af65ca840fc9cfe5a5b75793a61ce4a908be6b36c2613c3254c5c: Status 404 returned error can't find the container with id 2e78b0b2a61af65ca840fc9cfe5a5b75793a61ce4a908be6b36c2613c3254c5c Nov 03 07:17:22.690569 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.689333 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:17:22.690569 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.689416 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:17:22.690569 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.689455 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:17:22.690761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.690733 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:17:22.690823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.690800 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:17:22.690870 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.690837 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:17:22.758333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.758307 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt"] Nov 03 07:17:22.764404 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.764363 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:17:22.764509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.764430 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:17:22.764509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.764469 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:17:22.923649 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:22.923619 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e3e173c-8bab-4805-8a40-fd77e4941d22" path="/var/lib/kubelet/pods/6e3e173c-8bab-4805-8a40-fd77e4941d22/volumes" Nov 03 07:17:23.298747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.297481 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" event={"ID":"49c96119-0bed-477e-9992-eb73a9761fca","Type":"ContainerStarted","Data":"a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56"} Nov 03 07:17:23.298747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.297546 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" event={"ID":"49c96119-0bed-477e-9992-eb73a9761fca","Type":"ContainerStarted","Data":"c7e4d1c586e50f9eef8fd3a6ebf7994179a5b8c3836a29907d3fcf5cbe9db084"} Nov 03 07:17:23.304736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.300946 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" event={"ID":"7e54c4bd-d566-42ab-8613-67b296c03c18","Type":"ContainerStarted","Data":"0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71"} Nov 03 07:17:23.304736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.301007 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" event={"ID":"7e54c4bd-d566-42ab-8613-67b296c03c18","Type":"ContainerStarted","Data":"2e78b0b2a61af65ca840fc9cfe5a5b75793a61ce4a908be6b36c2613c3254c5c"} Nov 03 07:17:23.315736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.314547 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" event={"ID":"ec563540-af7c-4fe7-8b7f-cc4be7102ab8","Type":"ContainerStarted","Data":"4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb"} Nov 03 07:17:23.315736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.314612 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" event={"ID":"ec563540-af7c-4fe7-8b7f-cc4be7102ab8","Type":"ContainerStarted","Data":"043a58359558a13463327b243809e3d49dbebb84388f48f365dfd2f1f5f30c47"} Nov 03 07:17:23.327739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.317933 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" event={"ID":"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e","Type":"ContainerStarted","Data":"bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc"} Nov 03 07:17:23.327739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.317991 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" event={"ID":"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e","Type":"ContainerStarted","Data":"f529d09248dd5e8498b43f4700dfa3406774ee2f8f13e1a0fbf9d0d01f4429e1"} Nov 03 07:17:23.371662 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.371630 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:23.372082 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.372058 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:23.372242 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.372218 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 07:17:23.583737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.582749 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" podStartSLOduration=2.582729631 podStartE2EDuration="2.582729631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:17:23.430432009 +0000 UTC m=+1098.622973761" watchObservedRunningTime="2026-11-03 07:17:23.582729631 +0000 UTC m=+1098.775271379" Nov 03 07:17:23.733308 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.733240 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" podStartSLOduration=2.733219197 podStartE2EDuration="2.733219197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:17:23.597201203 +0000 UTC m=+1098.789742998" watchObservedRunningTime="2026-11-03 07:17:23.733219197 +0000 UTC m=+1098.925760950" Nov 03 07:17:23.745766 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:23.733651 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" podStartSLOduration=2.733640084 podStartE2EDuration="2.733640084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:17:23.730029879 +0000 UTC m=+1098.922571630" watchObservedRunningTime="2026-11-03 07:17:23.733640084 +0000 UTC m=+1098.926181838" Nov 03 07:17:24.241973 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:24.241926 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:24.245827 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:24.245756 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:24.246128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:24.246109 4409 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 07:17:24.246298 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:24.246273 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 07:17:24.246872 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:24.246833 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 07:17:24.247056 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:24.247031 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" podUID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 07:17:24.259556 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:24.259374 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:24.259556 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:24.259426 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 07:17:24.259556 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:24.259491 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:24.381740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:24.380141 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:24.381740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:24.380207 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 07:17:25.103029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:25.101833 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:17:25.185739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:25.184829 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:17:25.285741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:25.275380 4409 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 07:17:25.285741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:25.275490 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 07:17:25.285741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:25.281835 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 07:17:25.285741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:25.281904 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" podUID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 07:17:25.285741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:25.282206 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:25.285741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:25.282237 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 07:17:25.388026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:25.387932 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:25.388026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:25.387988 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 07:17:26.241742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:26.240259 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 07:17:26.241742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:26.240322 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" podUID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 07:17:26.251143 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:26.251112 4409 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 07:17:26.251613 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:26.251578 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 07:17:26.272737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:26.260080 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:26.272737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:26.260125 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 07:17:26.371742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:26.371229 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:26.371742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:26.371302 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 07:17:26.446768 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:26.445757 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Nov 03 07:17:27.239162 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:27.239123 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 07:17:27.249989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:27.239370 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" podUID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 07:17:27.249989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:27.244948 4409 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 07:17:27.249989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:27.244996 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 07:17:27.259070 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:27.259041 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:27.259280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:27.259255 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 07:17:27.369689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:27.369579 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:27.369689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:27.369634 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 07:17:28.099862 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:28.099809 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:17:28.232866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:28.232822 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:17:28.242254 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:28.242128 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 07:17:28.242254 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:28.242192 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" podUID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 07:17:28.246176 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:28.246067 4409 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 07:17:28.246176 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:28.246120 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 07:17:28.258265 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:28.258146 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:28.258265 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:28.258204 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 07:17:28.370288 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:28.370149 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:28.370288 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:28.370219 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 07:17:29.235762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:29.235473 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:17:29.236281 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:29.235936 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:17:29.243737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:29.240015 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 07:17:29.243737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:29.240061 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" podUID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 07:17:29.247736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:29.244972 4409 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 07:17:29.247736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:29.245010 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 07:17:29.268737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:29.258009 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:29.268737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:29.258053 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 07:17:29.365739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:29.361831 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" podStartSLOduration=8.361813432 podStartE2EDuration="8.361813432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:17:23.860505731 +0000 UTC m=+1099.053047494" watchObservedRunningTime="2026-11-03 07:17:29.361813432 +0000 UTC m=+1104.554355182" Nov 03 07:17:29.372739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:29.371058 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:29.372739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:29.371114 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 07:17:29.456078 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:29.456041 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:17:30.262757 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:30.262701 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 07:17:30.263342 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:30.263292 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 07:17:30.369867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:30.369826 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 03 07:17:30.370137 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:30.370108 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 03 07:17:30.599267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:30.593748 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:30.599267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:30.598561 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:30.603277 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:30.603252 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:17:31.269366 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:31.269334 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:31.269889 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:31.269755 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:31.270956 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:31.270933 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:17:31.296899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:31.296866 4409 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:17:31.297033 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:31.296916 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:17:31.309933 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:31.309903 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:31.310897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:31.310871 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:31.312280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:31.312258 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:17:31.374168 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:31.374143 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:31.374440 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:31.374423 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:31.375119 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:17:31.375102 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:17:38.774341 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:17:38.774305 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:18:06.645261 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.645225 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94"] Nov 03 07:18:06.650879 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.650859 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.758054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.758027 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-model-cache\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.758054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.758056 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf7l5\" (UniqueName: \"kubernetes.io/projected/12c21443-7e7e-46c0-835b-ad5cb444ddcb-kube-api-access-mf7l5\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.758302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.758083 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-dshm\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.758302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.758264 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tmp-dir\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.758417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.758304 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tls-certs\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.758491 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.758471 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-home\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.761167 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.761147 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9697f7\"/\"kube-root-ca.crt\"" Nov 03 07:18:06.761295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.761226 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9697f7\"/\"openshift-service-ca.crt\"" Nov 03 07:18:06.761343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.761312 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9697f7\"/\"distinct-w1-9697f7-kserve-self-signed-certs\"" Nov 03 07:18:06.761380 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.761365 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9697f7\"/\"default-dockercfg-5dnr4\"" Nov 03 07:18:06.764525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.764498 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94"] Nov 03 07:18:06.860349 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.860315 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tmp-dir\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.860543 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.860364 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tls-certs\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.860543 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.860513 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-home\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.860674 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.860563 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-model-cache\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.860674 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.860588 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf7l5\" (UniqueName: \"kubernetes.io/projected/12c21443-7e7e-46c0-835b-ad5cb444ddcb-kube-api-access-mf7l5\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.860674 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.860624 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-dshm\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.860872 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.860804 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tmp-dir\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.860872 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.860843 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-home\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.860976 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.860962 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-model-cache\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.862947 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.862924 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-dshm\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.863116 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.863099 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tls-certs\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.927507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.927438 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf7l5\" (UniqueName: \"kubernetes.io/projected/12c21443-7e7e-46c0-835b-ad5cb444ddcb-kube-api-access-mf7l5\") pod \"distinct-w1-9697f7-kserve-69ddff45bd-7rr94\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:06.962787 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:06.962757 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:07.377067 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:07.376865 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94"] Nov 03 07:18:07.379448 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:18:07.379417 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12c21443_7e7e_46c0_835b_ad5cb444ddcb.slice/crio-19ee1f5d4e9c12788c95651859c14359c10782bb77cb3a71f6d911948680dae8 WatchSource:0}: Error finding container 19ee1f5d4e9c12788c95651859c14359c10782bb77cb3a71f6d911948680dae8: Status 404 returned error can't find the container with id 19ee1f5d4e9c12788c95651859c14359c10782bb77cb3a71f6d911948680dae8 Nov 03 07:18:07.769782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:07.768845 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" event={"ID":"12c21443-7e7e-46c0-835b-ad5cb444ddcb","Type":"ContainerStarted","Data":"9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913"} Nov 03 07:18:07.769782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:07.768905 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" event={"ID":"12c21443-7e7e-46c0-835b-ad5cb444ddcb","Type":"ContainerStarted","Data":"19ee1f5d4e9c12788c95651859c14359c10782bb77cb3a71f6d911948680dae8"} Nov 03 07:18:07.893733 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:07.892950 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podStartSLOduration=1.892929377 podStartE2EDuration="1.892929377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:18:07.891116819 +0000 UTC m=+1143.083658569" watchObservedRunningTime="2026-11-03 07:18:07.892929377 +0000 UTC m=+1143.085471138" Nov 03 07:18:07.967736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:07.966798 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:08.849545 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:18:08.846906 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:18:16.963481 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:16.963450 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:16.976088 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:16.976069 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:17.860766 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:17.860739 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:18:38.774907 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:18:38.774872 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:18:53.963543 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:53.963507 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn"] Nov 03 07:18:53.969566 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:53.969532 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.043267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.043235 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-tmp-dir\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.043267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.043271 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lklsm\" (UniqueName: \"kubernetes.io/projected/952827fe-e2c2-4471-bce0-07be8470d6c7-kube-api-access-lklsm\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.043533 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.043434 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-home\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.043533 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.043470 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-model-cache\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.043648 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.043539 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952827fe-e2c2-4471-bce0-07be8470d6c7-tls-certs\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.043648 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.043610 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-dshm\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.069227 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.069187 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn"] Nov 03 07:18:54.073153 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.073133 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-3b41b9\"/\"distinct-w6-3b41b9-kserve-self-signed-certs\"" Nov 03 07:18:54.144452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.144416 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952827fe-e2c2-4471-bce0-07be8470d6c7-tls-certs\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.144611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.144532 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-dshm\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.144611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.144580 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-tmp-dir\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.144611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.144602 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lklsm\" (UniqueName: \"kubernetes.io/projected/952827fe-e2c2-4471-bce0-07be8470d6c7-kube-api-access-lklsm\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.144829 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.144769 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-home\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.144829 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.144811 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-model-cache\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.145078 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.145055 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-tmp-dir\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.145157 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.145122 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-home\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.145194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.145153 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-model-cache\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.146907 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.146884 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-dshm\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.147219 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.147201 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952827fe-e2c2-4471-bce0-07be8470d6c7-tls-certs\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.207856 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.207830 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lklsm\" (UniqueName: \"kubernetes.io/projected/952827fe-e2c2-4471-bce0-07be8470d6c7-kube-api-access-lklsm\") pod \"distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.285429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.285369 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:18:54.703915 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:54.703893 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn"] Nov 03 07:18:54.704864 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:18:54.704840 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod952827fe_e2c2_4471_bce0_07be8470d6c7.slice/crio-8c8c385381e51b75e474ac89912dff8bb6254b2ffd4cd451b5e218477559f1c5 WatchSource:0}: Error finding container 8c8c385381e51b75e474ac89912dff8bb6254b2ffd4cd451b5e218477559f1c5: Status 404 returned error can't find the container with id 8c8c385381e51b75e474ac89912dff8bb6254b2ffd4cd451b5e218477559f1c5 Nov 03 07:18:55.149367 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:55.149323 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" event={"ID":"952827fe-e2c2-4471-bce0-07be8470d6c7","Type":"ContainerStarted","Data":"7e6c6e500d8aabaed579380e260b301e01854705b70e8a304889deba742b9baf"} Nov 03 07:18:55.149883 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:55.149372 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" event={"ID":"952827fe-e2c2-4471-bce0-07be8470d6c7","Type":"ContainerStarted","Data":"8c8c385381e51b75e474ac89912dff8bb6254b2ffd4cd451b5e218477559f1c5"} Nov 03 07:18:55.271331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:55.271282 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podStartSLOduration=2.271268703 podStartE2EDuration="2.271268703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:18:55.268541774 +0000 UTC m=+1190.461083524" watchObservedRunningTime="2026-11-03 07:18:55.271268703 +0000 UTC m=+1190.463810453" Nov 03 07:18:55.285472 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:18:55.285451 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:19:04.286404 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:19:04.286374 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:19:04.298814 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:19:04.298790 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:19:05.020190 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:19:05.020158 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:19:05.023861 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:19:05.023842 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:19:05.213301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:19:05.213274 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:19:08.773269 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:19:08.773231 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:19:38.774408 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:19:38.774373 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:20:08.774840 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:20:08.774803 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:20:22.516182 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.516149 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg"] Nov 03 07:20:22.520248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.520228 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.549721 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.549693 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.549892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.549769 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xsr2\" (UniqueName: \"kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-kube-api-access-6xsr2\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.549892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.549859 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.550085 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.549990 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.550085 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.550039 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.550085 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.550063 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.550827 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.550182 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.550827 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.550735 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.550827 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.550758 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.627799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.627770 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg"] Nov 03 07:20:22.651706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.651678 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.651871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.651706 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.651871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.651795 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.651871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.651849 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.652046 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.651874 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.652046 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.651934 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.652046 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.652017 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xsr2\" (UniqueName: \"kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-kube-api-access-6xsr2\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.652184 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.652063 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.652243 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.652188 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.652305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.652276 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.652363 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.652304 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.652363 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.652346 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.652490 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.652411 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.652909 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.652889 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.654805 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.654782 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.655026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.655005 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.714692 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.714661 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.714806 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.714738 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xsr2\" (UniqueName: \"kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-kube-api-access-6xsr2\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:22.833237 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:22.833209 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:23.261678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:23.261655 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg"] Nov 03 07:20:23.263296 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:20:23.263276 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0484f1_4f69_4550_a16f_1ccc33b0c0f4.slice/crio-0c4be6fe04b5d5f9533cba7ca9b0c77d5b73cc5e27aaf8d89d9c2c837b8ec39d WatchSource:0}: Error finding container 0c4be6fe04b5d5f9533cba7ca9b0c77d5b73cc5e27aaf8d89d9c2c837b8ec39d: Status 404 returned error can't find the container with id 0c4be6fe04b5d5f9533cba7ca9b0c77d5b73cc5e27aaf8d89d9c2c837b8ec39d Nov 03 07:20:23.265622 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:23.265593 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:20:23.265687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:23.265655 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:20:23.265745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:23.265684 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:20:23.645007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:23.644971 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" event={"ID":"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4","Type":"ContainerStarted","Data":"f51160c629c14e4a6f145bccfd9dd00ff61fb6013838cf55f38cfc4334124302"} Nov 03 07:20:23.645391 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:23.645036 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" event={"ID":"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4","Type":"ContainerStarted","Data":"0c4be6fe04b5d5f9533cba7ca9b0c77d5b73cc5e27aaf8d89d9c2c837b8ec39d"} Nov 03 07:20:23.770003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:23.769958 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" podStartSLOduration=1.7699434269999998 podStartE2EDuration="1.769943427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:20:23.768871392 +0000 UTC m=+1278.961413142" watchObservedRunningTime="2026-11-03 07:20:23.769943427 +0000 UTC m=+1278.962485178" Nov 03 07:20:23.834159 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:23.834128 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:24.835960 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:24.835923 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 07:20:24.836388 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:24.835987 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 07:20:25.834448 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:25.834412 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 07:20:25.834622 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:25.834478 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 07:20:26.837745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:26.835912 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 07:20:26.837745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:26.835978 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 07:20:27.834168 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:27.834135 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 07:20:27.834361 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:27.834201 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 07:20:28.834294 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:28.834264 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 07:20:28.834673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:28.834326 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 07:20:29.834250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:29.834219 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 07:20:29.834419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:29.834274 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 07:20:30.985382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:30.985350 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf"] Nov 03 07:20:30.996703 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:30.996673 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.055384 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.055358 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf"] Nov 03 07:20:31.082828 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.082798 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-home\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.082980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.082843 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-dshm\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.082980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.082861 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf0fec39-da43-4132-af60-9948d7f78cf3-tls-certs\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.083058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.083006 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-model-cache\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.083058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.083035 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-tmp-dir\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.083124 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.083069 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsjg6\" (UniqueName: \"kubernetes.io/projected/cf0fec39-da43-4132-af60-9948d7f78cf3-kube-api-access-lsjg6\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.101229 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.101203 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"default-dockercfg-td9rn\"" Nov 03 07:20:31.101364 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.101295 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"distinct-2-w2-669d6b-kserve-self-signed-certs\"" Nov 03 07:20:31.101364 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.101306 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"kube-root-ca.crt\"" Nov 03 07:20:31.101364 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.101306 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-669d6b\"/\"openshift-service-ca.crt\"" Nov 03 07:20:31.184039 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.184005 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsjg6\" (UniqueName: \"kubernetes.io/projected/cf0fec39-da43-4132-af60-9948d7f78cf3-kube-api-access-lsjg6\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.184228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.184190 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-home\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.184228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.184221 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-dshm\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.184332 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.184247 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf0fec39-da43-4132-af60-9948d7f78cf3-tls-certs\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.184382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.184333 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-model-cache\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.184382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.184369 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-tmp-dir\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.184635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.184608 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-home\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.184792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.184697 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-model-cache\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.184792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.184778 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-tmp-dir\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.187135 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.187114 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-dshm\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.187365 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.187348 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf0fec39-da43-4132-af60-9948d7f78cf3-tls-certs\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.256955 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.256880 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsjg6\" (UniqueName: \"kubernetes.io/projected/cf0fec39-da43-4132-af60-9948d7f78cf3-kube-api-access-lsjg6\") pod \"distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.311889 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.311868 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:31.782447 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.782417 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf"] Nov 03 07:20:31.785306 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:20:31.785276 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf0fec39_da43_4132_af60_9948d7f78cf3.slice/crio-eb3e427f11e7602e13e7893ca10805aba730745d81770fedfbe76ef66314b13d WatchSource:0}: Error finding container eb3e427f11e7602e13e7893ca10805aba730745d81770fedfbe76ef66314b13d: Status 404 returned error can't find the container with id eb3e427f11e7602e13e7893ca10805aba730745d81770fedfbe76ef66314b13d Nov 03 07:20:31.877152 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.876823 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:20:31.877152 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:31.876871 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:32.125138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:32.125098 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:32.125761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:32.125706 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:32.126820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:32.126801 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:20:32.722598 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:32.722562 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" event={"ID":"cf0fec39-da43-4132-af60-9948d7f78cf3","Type":"ContainerStarted","Data":"f2fe42ea6c3d5a18f25672e09d72437b6f9aaac5715a1e0f32d3f18f1219354f"} Nov 03 07:20:32.722789 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:32.722614 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" event={"ID":"cf0fec39-da43-4132-af60-9948d7f78cf3","Type":"ContainerStarted","Data":"eb3e427f11e7602e13e7893ca10805aba730745d81770fedfbe76ef66314b13d"} Nov 03 07:20:32.843882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:32.843835 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podStartSLOduration=2.8438190629999998 podStartE2EDuration="2.843819063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:20:32.842392224 +0000 UTC m=+1288.034933977" watchObservedRunningTime="2026-11-03 07:20:32.843819063 +0000 UTC m=+1288.036360813" Nov 03 07:20:33.312531 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:33.312498 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:38.775908 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:20:38.775873 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:20:41.312223 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:41.312181 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:41.325457 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:41.325427 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:41.796924 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:41.796838 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:20:56.246781 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:56.246646 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:56.320250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:56.320209 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:20:56.467738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:56.466847 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 07:20:58.921569 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:58.921538 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm"] Nov 03 07:20:58.925262 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:58.925243 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:58.936318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:58.936292 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-model-cache\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:58.936411 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:58.936349 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tls-certs\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:58.936476 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:58.936456 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-dshm\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:58.936584 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:58.936566 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-home\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:58.936633 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:58.936601 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrff\" (UniqueName: \"kubernetes.io/projected/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-kube-api-access-qbrff\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:58.936832 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:58.936810 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tmp-dir\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.038342 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.038252 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-3b41b9\"/\"distinct-2-w6-3b41b9-kserve-self-signed-certs\"" Nov 03 07:20:59.039172 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.039109 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-model-cache\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.046511 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.046470 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tls-certs\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.047398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.047315 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-dshm\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.048074 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.048049 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-home\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.050032 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.049994 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrff\" (UniqueName: \"kubernetes.io/projected/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-kube-api-access-qbrff\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.050376 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.050352 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tmp-dir\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.051093 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.051071 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tmp-dir\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.051247 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.041192 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm"] Nov 03 07:20:59.051671 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.039811 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-model-cache\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.052140 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.052120 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-home\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.055390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.055366 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-dshm\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.056945 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.056921 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tls-certs\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.121101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.121080 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrff\" (UniqueName: \"kubernetes.io/projected/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-kube-api-access-qbrff\") pod \"distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.239830 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.239805 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:20:59.690619 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.690599 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm"] Nov 03 07:20:59.958542 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.958456 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" event={"ID":"ee0451f1-c74f-491f-8ae6-9e6786e8ff34","Type":"ContainerStarted","Data":"481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2"} Nov 03 07:20:59.958542 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:20:59.958504 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" event={"ID":"ee0451f1-c74f-491f-8ae6-9e6786e8ff34","Type":"ContainerStarted","Data":"e3704a7a7ea52cedaeb419dd05ef0f11dd33d2317755105d8884a4dde44656bc"} Nov 03 07:21:00.080408 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:00.080360 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podStartSLOduration=2.080347832 podStartE2EDuration="2.080347832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:00.077329599 +0000 UTC m=+1315.269871350" watchObservedRunningTime="2026-11-03 07:21:00.080347832 +0000 UTC m=+1315.272889582" Nov 03 07:21:00.240015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:00.239940 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:21:08.775839 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:21:08.775801 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:21:09.240489 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:09.240457 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:21:09.252897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:09.252874 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:21:10.024403 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:10.024376 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:21:22.905946 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:22.905913 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg"] Nov 03 07:21:22.909689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:22.909665 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.018961 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.018927 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4"] Nov 03 07:21:23.020881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.020852 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.020997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.020923 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.020997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.020969 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.021191 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.021174 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482ac46d-f7b3-45f9-9923-4a8205c11e41-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.021282 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.021263 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.021357 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.021343 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7wf\" (UniqueName: \"kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-kube-api-access-nw7wf\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.021402 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.021390 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.021516 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.021501 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.021556 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.021536 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.022774 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.022757 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc"] Nov 03 07:21:23.022886 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.022843 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.026624 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.026605 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.037761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.037735 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg"] Nov 03 07:21:23.124305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124275 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.124577 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124526 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.124751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124579 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.124751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124606 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.124751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124631 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.124751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124669 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482ac46d-f7b3-45f9-9923-4a8205c11e41-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.124751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124699 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a951fa97-ef86-4052-b96e-cf304f590ef6-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.125080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124765 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.125080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124815 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7wf\" (UniqueName: \"kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-kube-api-access-nw7wf\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.125080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124857 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.125080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124886 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.125080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124921 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4db9\" (UniqueName: \"kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-kube-api-access-s4db9\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.125080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124957 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.125080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.124981 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7ec919-d5c4-44e9-a498-82e604f21880-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.125080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125006 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.125080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125062 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.125564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125114 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.125564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125144 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.125564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125174 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.125564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125203 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.125564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125296 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.125564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125346 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.125564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125378 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bzw7\" (UniqueName: \"kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-kube-api-access-4bzw7\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.125564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125414 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.125564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125439 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.125564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125462 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.125564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.125494 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.126520 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.126495 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482ac46d-f7b3-45f9-9923-4a8205c11e41-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.126831 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.126810 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.127555 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.127529 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.127822 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.127797 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4"] Nov 03 07:21:23.129620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.128149 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.129620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.128517 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.129620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.129424 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc"] Nov 03 07:21:23.131406 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.131363 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.131653 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.131629 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.145941 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.145917 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9"] Nov 03 07:21:23.150839 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.150820 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.155526 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.155504 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5"] Nov 03 07:21:23.159841 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.159797 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.196450 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.196424 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.198011 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.197991 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7wf\" (UniqueName: \"kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-kube-api-access-nw7wf\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.225401 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.226569 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.226603 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.226622 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.226644 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.226675 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a951fa97-ef86-4052-b96e-cf304f590ef6-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.226767 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.226802 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.226832 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4db9\" (UniqueName: \"kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-kube-api-access-s4db9\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.226862 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7ec919-d5c4-44e9-a498-82e604f21880-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.226878 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.226919 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.226960 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.227006 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.227067 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.227087 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bzw7\" (UniqueName: \"kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-kube-api-access-4bzw7\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.227118 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.227141 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.227338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.227169 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.228670 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.228215 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9"] Nov 03 07:21:23.228670 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.228295 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.228670 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.228568 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.230603 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.230580 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.231280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.231259 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.231755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.231729 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5"] Nov 03 07:21:23.231911 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.231889 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.232007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.231953 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a951fa97-ef86-4052-b96e-cf304f590ef6-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.232091 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.232053 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.232319 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.232291 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.232553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.232534 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.232864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.232837 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.236634 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.236608 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.237482 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.237460 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.239920 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.239894 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7ec919-d5c4-44e9-a498-82e604f21880-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.241970 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.241950 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.296933 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.296768 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.303954 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.303640 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bzw7\" (UniqueName: \"kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-kube-api-access-4bzw7\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.304637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.304611 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4db9\" (UniqueName: \"kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-kube-api-access-s4db9\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.307054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.307030 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.328887 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.328857 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.328996 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.328904 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.328996 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.328934 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.329110 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.328996 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.329110 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329088 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkcgk\" (UniqueName: \"kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-kube-api-access-bkcgk\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.329215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329147 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.329215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329207 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.329318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329236 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.329318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329265 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.329318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329308 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.329473 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329342 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.329473 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329373 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.329473 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329465 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.329630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329526 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.329630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329569 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.329630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329600 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbr69\" (UniqueName: \"kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-kube-api-access-jbr69\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.329794 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329627 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.329794 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.329682 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.348190 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.348169 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:23.357498 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.357476 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:23.431063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431025 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.431177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431086 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.431177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431116 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.431177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431138 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.431177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431169 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkcgk\" (UniqueName: \"kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-kube-api-access-bkcgk\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.431401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431194 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.431401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431237 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.431401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431263 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.431401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431289 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.431401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431336 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.431401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431369 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.431752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431400 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.431752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431502 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.431752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431571 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.431752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431617 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.431752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431650 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbr69\" (UniqueName: \"kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-kube-api-access-jbr69\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.431752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431678 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.431752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.431753 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.432226 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.432203 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.432853 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.432827 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.432949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.432924 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.432990 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.432952 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.433218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.433200 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.433483 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.433450 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.434284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.434217 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.434284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.434221 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.434494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.434469 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.434566 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.434496 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.436030 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.435997 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.436443 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.436427 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.436959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.436942 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.438636 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.438621 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.497279 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.497248 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.497431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.497355 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkcgk\" (UniqueName: \"kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-kube-api-access-bkcgk\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.497576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.497557 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbr69\" (UniqueName: \"kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-kube-api-access-jbr69\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.501029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.501010 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:23.621744 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:21:23.621688 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482ac46d_f7b3_45f9_9923_4a8205c11e41.slice/crio-842118be2cb97644a40aef5205d1bbdeaae74a7d82021ce10dbc7066e4f796fd WatchSource:0}: Error finding container 842118be2cb97644a40aef5205d1bbdeaae74a7d82021ce10dbc7066e4f796fd: Status 404 returned error can't find the container with id 842118be2cb97644a40aef5205d1bbdeaae74a7d82021ce10dbc7066e4f796fd Nov 03 07:21:23.621889 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.621837 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg"] Nov 03 07:21:23.624412 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.624378 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:23.624480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.624457 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:23.624513 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.624495 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:23.744057 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.744026 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4"] Nov 03 07:21:23.747531 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:21:23.747502 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7ec919_d5c4_44e9_a498_82e604f21880.slice/crio-a04ff7dcb6a395ade9caf6f838ea1ca929ebfc7cb2c30732b94affb429ce3755 WatchSource:0}: Error finding container a04ff7dcb6a395ade9caf6f838ea1ca929ebfc7cb2c30732b94affb429ce3755: Status 404 returned error can't find the container with id a04ff7dcb6a395ade9caf6f838ea1ca929ebfc7cb2c30732b94affb429ce3755 Nov 03 07:21:23.750209 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.750181 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:23.750284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.750244 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:23.750284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.750274 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:23.759703 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.759677 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc"] Nov 03 07:21:23.761576 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:21:23.761544 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda951fa97_ef86_4052_b96e_cf304f590ef6.slice/crio-3f0e7323749e2165318b156b6ec5db06024e4b04662d408b257afab728caa3dc WatchSource:0}: Error finding container 3f0e7323749e2165318b156b6ec5db06024e4b04662d408b257afab728caa3dc: Status 404 returned error can't find the container with id 3f0e7323749e2165318b156b6ec5db06024e4b04662d408b257afab728caa3dc Nov 03 07:21:23.765667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.765633 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:23.765772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.765744 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:23.765813 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.765789 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:23.778703 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.778528 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:23.781364 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:23.781345 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:24.098285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.098250 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" event={"ID":"a951fa97-ef86-4052-b96e-cf304f590ef6","Type":"ContainerStarted","Data":"9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c"} Nov 03 07:21:24.098893 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.098855 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" event={"ID":"a951fa97-ef86-4052-b96e-cf304f590ef6","Type":"ContainerStarted","Data":"3f0e7323749e2165318b156b6ec5db06024e4b04662d408b257afab728caa3dc"} Nov 03 07:21:24.110149 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.110113 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" event={"ID":"4f7ec919-d5c4-44e9-a498-82e604f21880","Type":"ContainerStarted","Data":"419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a"} Nov 03 07:21:24.110291 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.110176 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" event={"ID":"4f7ec919-d5c4-44e9-a498-82e604f21880","Type":"ContainerStarted","Data":"a04ff7dcb6a395ade9caf6f838ea1ca929ebfc7cb2c30732b94affb429ce3755"} Nov 03 07:21:24.113970 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.113944 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" event={"ID":"482ac46d-f7b3-45f9-9923-4a8205c11e41","Type":"ContainerStarted","Data":"1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0"} Nov 03 07:21:24.114085 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.113994 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" event={"ID":"482ac46d-f7b3-45f9-9923-4a8205c11e41","Type":"ContainerStarted","Data":"842118be2cb97644a40aef5205d1bbdeaae74a7d82021ce10dbc7066e4f796fd"} Nov 03 07:21:24.196151 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.196123 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9"] Nov 03 07:21:24.203142 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.203102 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:24.203271 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.203200 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:24.203271 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.203248 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:24.222438 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.222414 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5"] Nov 03 07:21:24.224326 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:21:24.224304 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfdadfa_4e8d_49ca_a1bf_4a3cd6a897b8.slice/crio-df4a4f39c14b39f484410a4168147ca6766332a299cfa1dee5de30ad7d164ee7 WatchSource:0}: Error finding container df4a4f39c14b39f484410a4168147ca6766332a299cfa1dee5de30ad7d164ee7: Status 404 returned error can't find the container with id df4a4f39c14b39f484410a4168147ca6766332a299cfa1dee5de30ad7d164ee7 Nov 03 07:21:24.225781 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.225742 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podStartSLOduration=2.225696698 podStartE2EDuration="2.225696698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:24.222760128 +0000 UTC m=+1339.415301878" watchObservedRunningTime="2026-11-03 07:21:24.225696698 +0000 UTC m=+1339.418238447" Nov 03 07:21:24.227029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.226927 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:24.228328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.228140 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 07:21:24.228328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.228179 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 07:21:24.228956 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.228926 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:24.229096 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.228992 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:24.229096 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.229026 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:21:24.352241 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.352074 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podStartSLOduration=2.352054764 podStartE2EDuration="2.352054764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:24.346776794 +0000 UTC m=+1339.539318547" watchObservedRunningTime="2026-11-03 07:21:24.352054764 +0000 UTC m=+1339.544596516" Nov 03 07:21:24.360190 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.359699 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:24.360705 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.360685 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:24.362835 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.362792 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:24.362991 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.362961 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:24.363114 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.363055 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:24.363114 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:24.363079 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 07:21:25.154826 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.154785 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" event={"ID":"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8","Type":"ContainerStarted","Data":"985dc45c9c9fa438a8e85db33f2778cd91dcb45047821b20559e6c576a597ba4"} Nov 03 07:21:25.161852 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.161806 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" event={"ID":"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8","Type":"ContainerStarted","Data":"df4a4f39c14b39f484410a4168147ca6766332a299cfa1dee5de30ad7d164ee7"} Nov 03 07:21:25.172739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.169422 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" event={"ID":"702dd12a-8cf6-4daa-9530-8f85b6a8d01c","Type":"ContainerStarted","Data":"bb2e13d35f6f0812aff7aee30f81bcc135989268a8ec294342cdbeed6c22c6ce"} Nov 03 07:21:25.172996 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.172971 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" event={"ID":"702dd12a-8cf6-4daa-9530-8f85b6a8d01c","Type":"ContainerStarted","Data":"9e474247c348cfed286958a24dbcf335ad290a863771a988bbc258f3567e91a0"} Nov 03 07:21:25.243743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.236986 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 07:21:25.244087 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.244039 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 07:21:25.290750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.288243 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podStartSLOduration=3.288224691 podStartE2EDuration="3.288224691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:24.492053884 +0000 UTC m=+1339.684595652" watchObservedRunningTime="2026-11-03 07:21:25.288224691 +0000 UTC m=+1340.480766442" Nov 03 07:21:25.369738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.348580 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:25.369738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.348636 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 07:21:25.369738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.362131 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:25.369738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.362185 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:25.516831 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.511114 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podStartSLOduration=2.511093465 podStartE2EDuration="2.511093465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:25.307083091 +0000 UTC m=+1340.499624824" watchObservedRunningTime="2026-11-03 07:21:25.511093465 +0000 UTC m=+1340.703635237" Nov 03 07:21:25.539753 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.533078 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podStartSLOduration=2.533060832 podStartE2EDuration="2.533060832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:21:25.434942493 +0000 UTC m=+1340.627484245" watchObservedRunningTime="2026-11-03 07:21:25.533060832 +0000 UTC m=+1340.725602583" Nov 03 07:21:25.781693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.781589 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:25.782013 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.781964 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 07:21:25.782505 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.782484 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:25.784137 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.784119 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:25.785246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.785228 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:25.785512 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:25.785487 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 07:21:26.149740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.148847 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.232742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.230860 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 07:21:26.232742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.230919 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 07:21:26.326760 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.324809 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.359654 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.359616 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:26.359943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.359913 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:26.360386 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.360367 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:26.360519 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.360500 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 07:21:26.435901 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.435803 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.459856 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.459796 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.780795 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.780054 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:26.780795 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.780113 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 07:21:26.784735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.783995 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:26.784735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.784045 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 07:21:26.796736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.794301 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:26.861845 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:26.861796 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 07:21:27.019739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.018809 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.152703 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.150804 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.152703 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.151246 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:21:27.223739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.222902 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 07:21:27.230104 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.230073 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 07:21:27.230369 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.230341 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 07:21:27.329751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.328827 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.329751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.329298 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:21:27.370745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.356472 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:27.370745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.356529 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 07:21:27.430797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.404326 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:27.439842 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.439797 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:27.579757 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.493813 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.579757 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.579264 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:21:27.819653 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.819610 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:27.819965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.819928 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 07:21:27.820158 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.820141 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:27.820305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.820286 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 07:21:27.861557 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.861515 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.862210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.862185 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:21:27.886742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.864367 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:27.888189 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:27.888163 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:21:28.052236 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.050269 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Nov 03 07:21:28.062751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.061540 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:28.064882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.064852 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:21:28.182923 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.182818 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:28.257977 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.257934 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:28.263257 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.263211 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:21:28.310764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.309179 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 07:21:28.310764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.309238 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 07:21:28.407742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.367071 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:28.409482 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.380662 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:28.409753 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.409698 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:28.409913 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.393558 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:28.410029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.410009 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 07:21:28.601744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.580046 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:28.999823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.977899 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:28.999823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:28.977973 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 07:21:29.095800 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.089891 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:21:29.179079 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.097828 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 07:21:29.179079 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.163316 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:29.179079 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.163362 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 07:21:29.248814 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.234068 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 07:21:29.295268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.295160 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 07:21:29.391924 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.391879 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:29.392285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.392258 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:29.493965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.471394 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:29.493965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.471458 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 07:21:29.498903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.498870 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:29.557934 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.557894 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:29.558181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.532987 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 07:21:29.558308 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.558288 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 07:21:29.604005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.603942 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6" podUID="6b36e4ad-8060-4a62-b8c2-7468072396ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 03 07:21:29.756745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.744014 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 07:21:29.868797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.828671 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:29.868797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:29.828781 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 07:21:30.037741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.026696 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:30.037741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.026783 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 07:21:30.090739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.090175 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:30.274798 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:21:30.271118 4409 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.337s" Nov 03 07:21:30.274798 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.271214 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:21:30.346328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.346288 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:30.383741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.382388 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 07:21:30.383741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.382448 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 07:21:30.439785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.438861 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:21:30.443194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.440537 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:21:30.467792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.467177 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:30.502750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.501903 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:30.502750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.502006 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 07:21:30.505240 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.505213 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:30.505488 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.505451 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:30.509970 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.509941 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:30.685740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.658470 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:21:30.730739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.708690 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:21:30.757071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.757028 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:30.829907 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.829862 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:30.830221 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.830169 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 07:21:30.838702 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.834122 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:30.855247 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:30.855207 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 07:21:31.202962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.202913 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.283743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.274492 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 07:21:31.283743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.274551 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 07:21:31.363128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.363080 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.400743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.378765 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:31.400743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.378826 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:31.400743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.378912 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:31.400743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.378934 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 07:21:31.474262 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.471226 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.524741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.510811 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.738803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.720368 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.759839 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.759795 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:31.788694 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.788661 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:31.795819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.795784 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 07:21:31.796010 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.790366 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:31.796142 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:31.796123 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 07:21:32.204739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.203786 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 07:21:32.232746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.232622 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 07:21:32.232746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.232693 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 07:21:32.326739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.326653 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.356079 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.356043 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:32.357117 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.357085 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 07:21:32.362860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.362809 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:32.363065 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.363010 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:32.366908 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.366788 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.367265 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.367233 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:21:32.396155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.396126 4409 patch_prober.go:29] interesting pod/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:21:32.396978 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.396774 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.471896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.471799 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.473469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.473448 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:21:32.474454 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.472107 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:21:32.522917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.522875 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.722535 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.722440 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.723637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.723613 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:21:32.793739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.763871 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:32.794248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.788087 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:32.794663 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.794635 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 07:21:32.794826 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.788372 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:32.794940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:32.794921 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 07:21:33.208879 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.208834 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:33.209557 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.209531 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:21:33.209785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.209758 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:33.240864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.240820 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:33.241114 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.241086 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 07:21:33.241199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.241132 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 07:21:33.351867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.351832 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:33.352137 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.352111 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 07:21:33.360073 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.360040 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:33.361409 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.361374 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:33.370353 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.370322 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:33.480014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.479911 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 07:21:33.481155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.481106 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:33.483768 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.482638 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:21:33.569094 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.569043 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:33.732184 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.732084 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:33.798392 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.798358 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:33.798652 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.798620 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 07:21:33.799156 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.799130 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:33.799297 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.799279 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 07:21:33.799468 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:33.799451 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:34.228741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.217300 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:34.229740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.229687 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 07:21:34.229949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.229919 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 07:21:34.390568 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.390533 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:34.394003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.393969 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 07:21:34.394305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.394282 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:34.394996 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.394975 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:34.454871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.454827 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:34.488909 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.488810 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:34.489995 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.489965 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:34.621866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.621810 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:34.780572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.780486 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:34.817750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.815965 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 07:21:34.817750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.816511 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:34.817750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.816550 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 07:21:34.817750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.817364 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:21:34.822275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.822247 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:34.822451 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:34.822432 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 07:21:35.007735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.007172 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:21:35.221284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.221239 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:35.235354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.235321 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 07:21:35.235580 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.235555 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 07:21:35.331750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.325676 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Nov 03 07:21:35.362193 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.361387 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 07:21:35.362193 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.361447 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 07:21:35.362193 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.361855 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:35.362193 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.361895 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:35.365357 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.364323 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:21:35.394626 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.394588 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:35.466770 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.464210 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:21:35.501792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.501675 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:21:35.508702 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.508663 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:21:35.534737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.531355 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:21:35.601328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.601296 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:21:35.788922 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.785343 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 07:21:35.832856 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.832810 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 07:21:35.854756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.810032 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 07:21:35.855105 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:35.855064 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 07:21:36.358474 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:36.358439 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 07:21:36.358767 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:36.358697 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 07:21:36.435084 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:36.435052 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:36.441557 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:36.441402 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:36.450059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:36.449990 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:21:36.636794 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:36.630685 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:36.636794 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:36.631889 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:36.656180 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:36.656131 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:21:37.608846 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:37.599917 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:37.612561 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:37.612539 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:37.613931 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:37.613903 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:21:37.638217 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:37.637738 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:37.639924 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:37.638970 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:37.640846 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:37.640100 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:21:37.780659 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:37.780622 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:21:37.780959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:37.780928 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:21:38.768388 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:38.768354 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:38.769911 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:38.769887 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:38.771505 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:38.771486 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:21:38.779349 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:21:38.779318 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:21:55.794738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:21:55.791787 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:08.779278 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:22:08.779238 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:22:24.203823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.201204 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z"] Nov 03 07:22:24.233646 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.233610 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.292367 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.292328 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z"] Nov 03 07:22:24.352825 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.352771 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7867\" (UniqueName: \"kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-kube-api-access-g7867\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.353122 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.353093 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.353323 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.353303 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.353521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.353504 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.353752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.353678 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.353930 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.353912 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.354136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.354119 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.380196 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.380162 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.387668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.381175 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.453741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.417971 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:24.532788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.522735 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7867\" (UniqueName: \"kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-kube-api-access-g7867\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.532788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.522893 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.532788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.523052 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.532788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.523216 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.532788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.523255 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.532788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.526884 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.532788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.530876 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:24.532788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.532264 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.551043 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.550891 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.552990 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.551462 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.553351 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.553329 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.553693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.553673 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.553990 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.552295 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.554199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.553948 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 07:22:24.661962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.552060 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.661962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.557987 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.852802 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.738245 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:24.852802 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.782023 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:24.852802 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.823498 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:24.852802 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.832470 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.862735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.853506 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.888743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.875927 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:24.949741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:24.924449 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7867\" (UniqueName: \"kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-kube-api-access-g7867\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:25.092911 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.092858 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.170599 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.170507 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:25.592610 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.537744 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.728469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.623045 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 07:22:25.728469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.720160 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:22:25.728469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.720217 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:22:25.728469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.637950 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.728469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.721063 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.728469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.722197 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:22:25.850260 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.823545 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.850260 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.823662 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:25.850260 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.825361 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 07:22:25.850260 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.825650 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:22:25.850260 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.825704 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:22:25.850260 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:25.825794 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:22:26.108836 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:26.096585 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:26.108836 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:26.096981 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:22:26.460892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:26.441972 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:26.863770 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:26.863704 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:26.893767 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:26.880288 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:27.051750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:26.869118 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 07:22:27.051750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:26.870192 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 07:22:27.051750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:26.870215 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:27.051750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:26.869083 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:27.102747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:27.097356 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:27.923146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:27.922866 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:28.021963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.010546 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:28.089986 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.086030 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:28.089986 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.086064 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 07:22:28.089986 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.086089 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 07:22:28.089986 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.086106 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:28.089986 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.086117 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:28.100422 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.100379 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:28.139882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.139843 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z"] Nov 03 07:22:28.143128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.143078 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:22:28.175253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.175167 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:22:28.331563 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.331533 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:22:28.333327 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.333305 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:22:28.338741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.336125 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:22:28.338741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.337776 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:22:28.351500 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.351450 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:28.351858 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.351829 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:28.353514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.353484 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:28.437017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:28.436014 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:22:29.341322 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:29.341285 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" event={"ID":"27d13c1e-7bc8-4579-9dd7-80c31d00cad2","Type":"ContainerStarted","Data":"f3d0ec3d40d1f3c5bdf9a26005627dc02c48692127afc44cb5c2c40671311e02"} Nov 03 07:22:29.341801 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:29.341386 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" event={"ID":"27d13c1e-7bc8-4579-9dd7-80c31d00cad2","Type":"ContainerStarted","Data":"8f45f1b5d68fbd292ff0acd53a726f00bc63c4c7f28cf30cd65c182f77b4209c"} Nov 03 07:22:29.493232 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:29.493175 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" podStartSLOduration=5.493159675 podStartE2EDuration="5.493159675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:29.49211629 +0000 UTC m=+1404.684658041" watchObservedRunningTime="2026-11-03 07:22:29.493159675 +0000 UTC m=+1404.685701425" Nov 03 07:22:30.173747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:30.173345 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:30.173747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:30.173467 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:22:30.173747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:30.173514 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" podUID="27d13c1e-7bc8-4579-9dd7-80c31d00cad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 07:22:31.181738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:31.180051 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:22:31.181738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:31.180114 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" podUID="27d13c1e-7bc8-4579-9dd7-80c31d00cad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 07:22:32.181160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:32.181118 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:22:32.181470 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:32.181438 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" podUID="27d13c1e-7bc8-4579-9dd7-80c31d00cad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 07:22:33.179062 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:33.179027 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:22:33.180010 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:33.179595 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" podUID="27d13c1e-7bc8-4579-9dd7-80c31d00cad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 07:22:34.179755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:34.172003 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:22:34.179755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:34.172059 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" podUID="27d13c1e-7bc8-4579-9dd7-80c31d00cad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 07:22:35.171975 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:35.171939 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 07:22:35.172147 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:35.172006 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" podUID="27d13c1e-7bc8-4579-9dd7-80c31d00cad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 07:22:36.240841 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:36.240809 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:36.241933 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:36.241902 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:36.243909 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:36.243889 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:22:38.830690 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:22:38.830646 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:22:39.336919 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.336888 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z"] Nov 03 07:22:39.341373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.341342 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.457401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.457366 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z"] Nov 03 07:22:39.522492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.522453 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.522664 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.522504 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.522664 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.522582 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.522762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.522693 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pv5t\" (UniqueName: \"kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-kube-api-access-6pv5t\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.522811 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.522795 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6dbc02a-e589-4172-8071-272bd1a343d4-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.522875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.522864 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.522963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.522950 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.523019 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.523005 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.523061 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.523052 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625039 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.624944 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625039 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.624996 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.625085 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.625124 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.625152 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.625235 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pv5t\" (UniqueName: \"kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-kube-api-access-6pv5t\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.625267 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6dbc02a-e589-4172-8071-272bd1a343d4-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.625361 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.625388 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-data\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.625412 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-credential-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.625422 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.625534 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-socket\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.625650 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-certs\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.625970 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.625948 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6dbc02a-e589-4172-8071-272bd1a343d4-istiod-ca-cert\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.627619 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.627596 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-envoy\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.628024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.628004 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-podinfo\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.696462 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.696425 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-token\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.697060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.697036 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pv5t\" (UniqueName: \"kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-kube-api-access-6pv5t\") pod \"e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:39.956272 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:39.956177 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:40.453941 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:22:40.453903 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dbc02a_e589_4172_8071_272bd1a343d4.slice/crio-ff3302dab3135cb846639564bc9af13333ea077125adea78d456ed92d0c37924 WatchSource:0}: Error finding container ff3302dab3135cb846639564bc9af13333ea077125adea78d456ed92d0c37924: Status 404 returned error can't find the container with id ff3302dab3135cb846639564bc9af13333ea077125adea78d456ed92d0c37924 Nov 03 07:22:40.457633 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.457598 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:40.457787 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.457661 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:40.457787 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.457691 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:40.457927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.457793 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z"] Nov 03 07:22:40.469761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.463256 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd"] Nov 03 07:22:40.506595 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.506563 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.578840 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.576838 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd"] Nov 03 07:22:40.578840 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.577255 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-67db4bc0-openshift-default-dockercfg-bgxk4\"" Nov 03 07:22:40.673823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.658817 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-envoy\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.673823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.658865 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-data\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.673823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.658903 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-credential-socket\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.673823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.658961 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f95955-229f-446c-ac9f-2fab451a3889-istio-podinfo\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.673823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.659021 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f95955-229f-446c-ac9f-2fab451a3889-istiod-ca-cert\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.673823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.659092 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-certs\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.673823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.659159 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-istio-token\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.673823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.659192 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-socket\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.673823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.659238 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtkqx\" (UniqueName: \"kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-kube-api-access-gtkqx\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.770743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.769451 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-envoy\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.770743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.769644 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-data\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.770743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.769763 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-credential-socket\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.770743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.769850 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f95955-229f-446c-ac9f-2fab451a3889-istio-podinfo\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.770743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.769919 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f95955-229f-446c-ac9f-2fab451a3889-istiod-ca-cert\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.770743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.770009 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-certs\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.770743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.770093 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-istio-token\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.770743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.770121 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-socket\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.770743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.770173 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtkqx\" (UniqueName: \"kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-kube-api-access-gtkqx\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.778492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.776429 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f95955-229f-446c-ac9f-2fab451a3889-istiod-ca-cert\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.778492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.776704 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-data\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.778492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.777111 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-credential-socket\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.822001 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.787500 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-certs\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.822001 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.787787 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-socket\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.856704 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.822489 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f95955-229f-446c-ac9f-2fab451a3889-istio-podinfo\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.936404 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.933889 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-envoy\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.936404 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.936053 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-istio-token\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:40.937488 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:40.936691 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtkqx\" (UniqueName: \"kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-kube-api-access-gtkqx\") pod \"e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:41.076311 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:41.076277 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:41.524387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:41.523403 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" event={"ID":"f6dbc02a-e589-4172-8071-272bd1a343d4","Type":"ContainerStarted","Data":"c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4"} Nov 03 07:22:41.524387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:41.523473 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" event={"ID":"f6dbc02a-e589-4172-8071-272bd1a343d4","Type":"ContainerStarted","Data":"ff3302dab3135cb846639564bc9af13333ea077125adea78d456ed92d0c37924"} Nov 03 07:22:41.548551 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:41.548513 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd"] Nov 03 07:22:41.626816 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:41.624606 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:41.626816 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:41.624730 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:41.626816 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:41.624834 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:41.956398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:41.956361 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:42.531881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:42.531842 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" event={"ID":"94f95955-229f-446c-ac9f-2fab451a3889","Type":"ContainerStarted","Data":"5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c"} Nov 03 07:22:42.532376 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:42.531912 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" event={"ID":"94f95955-229f-446c-ac9f-2fab451a3889","Type":"ContainerStarted","Data":"fd36e457b1dcdd56bbaeb6158361296e2e729570de383b74816146fe5ce345f9"} Nov 03 07:22:42.682929 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:42.682868 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" podStartSLOduration=3.682847775 podStartE2EDuration="3.682847775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:41.671418323 +0000 UTC m=+1416.863960072" watchObservedRunningTime="2026-11-03 07:22:42.682847775 +0000 UTC m=+1417.875389526" Nov 03 07:22:42.958914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:42.958880 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:22:42.959177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:42.958944 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" podUID="f6dbc02a-e589-4172-8071-272bd1a343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 07:22:43.088084 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:43.086772 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:43.094837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:43.094810 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:43.218272 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:43.218141 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" podStartSLOduration=3.218056621 podStartE2EDuration="3.218056621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:42.688129136 +0000 UTC m=+1417.880670888" watchObservedRunningTime="2026-11-03 07:22:43.218056621 +0000 UTC m=+1418.410598372" Nov 03 07:22:43.537983 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:43.537904 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:43.538799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:43.538783 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:22:43.957008 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:43.956969 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:22:43.957208 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:43.957023 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" podUID="f6dbc02a-e589-4172-8071-272bd1a343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 07:22:44.957132 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:44.957093 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:22:44.957562 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:44.957154 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" podUID="f6dbc02a-e589-4172-8071-272bd1a343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 07:22:45.957431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:45.957394 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:22:45.958138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:45.958105 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" podUID="f6dbc02a-e589-4172-8071-272bd1a343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 07:22:46.957251 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:46.957215 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 07:22:46.957422 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:46.957272 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" podUID="f6dbc02a-e589-4172-8071-272bd1a343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 07:22:48.417316 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:48.417282 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:48.418317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:48.418268 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:48.419241 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:48.419220 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:22:48.969473 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:48.969437 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf"] Nov 03 07:22:48.973767 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:48.973749 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" Nov 03 07:22:49.029454 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.029416 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-33a46b37\"" Nov 03 07:22:49.029571 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.029546 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-33a46b37-dockercfg-q72v9\"" Nov 03 07:22:49.032600 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.032580 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf"] Nov 03 07:22:49.050235 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.050205 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf\" (UID: \"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" Nov 03 07:22:49.050362 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.050256 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86xq8\" (UniqueName: \"kubernetes.io/projected/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-kube-api-access-86xq8\") pod \"payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf\" (UID: \"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" Nov 03 07:22:49.165775 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.152552 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf\" (UID: \"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" Nov 03 07:22:49.165775 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.152601 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86xq8\" (UniqueName: \"kubernetes.io/projected/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-kube-api-access-86xq8\") pod \"payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf\" (UID: \"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" Nov 03 07:22:49.165775 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.153757 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf\" (UID: \"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" Nov 03 07:22:49.245821 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.238785 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml"] Nov 03 07:22:49.245821 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.243891 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" Nov 03 07:22:49.248752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.246294 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml"] Nov 03 07:22:49.348259 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.341754 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86xq8\" (UniqueName: \"kubernetes.io/projected/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-kube-api-access-86xq8\") pod \"payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf\" (UID: \"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" Nov 03 07:22:49.364558 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.364532 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71b9dc9-00d9-43bd-8020-503fe826d1cc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml\" (UID: \"e71b9dc9-00d9-43bd-8020-503fe826d1cc\") " pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" Nov 03 07:22:49.364895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.364859 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrs8k\" (UniqueName: \"kubernetes.io/projected/e71b9dc9-00d9-43bd-8020-503fe826d1cc-kube-api-access-hrs8k\") pod \"payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml\" (UID: \"e71b9dc9-00d9-43bd-8020-503fe826d1cc\") " pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" Nov 03 07:22:49.465976 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.465947 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrs8k\" (UniqueName: \"kubernetes.io/projected/e71b9dc9-00d9-43bd-8020-503fe826d1cc-kube-api-access-hrs8k\") pod \"payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml\" (UID: \"e71b9dc9-00d9-43bd-8020-503fe826d1cc\") " pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" Nov 03 07:22:49.466403 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.466052 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71b9dc9-00d9-43bd-8020-503fe826d1cc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml\" (UID: \"e71b9dc9-00d9-43bd-8020-503fe826d1cc\") " pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" Nov 03 07:22:49.466765 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.466702 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71b9dc9-00d9-43bd-8020-503fe826d1cc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml\" (UID: \"e71b9dc9-00d9-43bd-8020-503fe826d1cc\") " pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" Nov 03 07:22:49.529307 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.529252 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrs8k\" (UniqueName: \"kubernetes.io/projected/e71b9dc9-00d9-43bd-8020-503fe826d1cc-kube-api-access-hrs8k\") pod \"payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml\" (UID: \"e71b9dc9-00d9-43bd-8020-503fe826d1cc\") " pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" Nov 03 07:22:49.587210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.587186 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" Nov 03 07:22:49.633689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.633663 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" Nov 03 07:22:49.970133 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:49.970105 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf"] Nov 03 07:22:49.970857 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:22:49.970831 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf62d1fb_e609_40b2_b232_5c2ed3eefcdc.slice/crio-5d1b09509336aeb70a85a5cfe92d13dc6f0d5dc8c2b4ce25c3cdd603e038c148 WatchSource:0}: Error finding container 5d1b09509336aeb70a85a5cfe92d13dc6f0d5dc8c2b4ce25c3cdd603e038c148: Status 404 returned error can't find the container with id 5d1b09509336aeb70a85a5cfe92d13dc6f0d5dc8c2b4ce25c3cdd603e038c148 Nov 03 07:22:50.075621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:50.075594 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml"] Nov 03 07:22:50.076619 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:22:50.076582 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71b9dc9_00d9_43bd_8020_503fe826d1cc.slice/crio-6e951e89c5da8e8871ac0914467fb4c800799c29eef09a3bb795877e764971de WatchSource:0}: Error finding container 6e951e89c5da8e8871ac0914467fb4c800799c29eef09a3bb795877e764971de: Status 404 returned error can't find the container with id 6e951e89c5da8e8871ac0914467fb4c800799c29eef09a3bb795877e764971de Nov 03 07:22:50.591518 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:50.591478 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" event={"ID":"e71b9dc9-00d9-43bd-8020-503fe826d1cc","Type":"ContainerStarted","Data":"06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541"} Nov 03 07:22:50.592000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:50.591550 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" event={"ID":"e71b9dc9-00d9-43bd-8020-503fe826d1cc","Type":"ContainerStarted","Data":"6e951e89c5da8e8871ac0914467fb4c800799c29eef09a3bb795877e764971de"} Nov 03 07:22:50.592000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:50.591577 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" Nov 03 07:22:50.593317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:50.593293 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" event={"ID":"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc","Type":"ContainerStarted","Data":"1cdaceb91a5e4c237ea738dc4bbb3209685cd20124572d0f10249f110d93ea41"} Nov 03 07:22:50.593447 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:50.593331 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" event={"ID":"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc","Type":"ContainerStarted","Data":"5d1b09509336aeb70a85a5cfe92d13dc6f0d5dc8c2b4ce25c3cdd603e038c148"} Nov 03 07:22:50.593447 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:50.593366 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" Nov 03 07:22:50.713604 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:50.713556 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" podStartSLOduration=1.713544024 podStartE2EDuration="1.713544024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:50.710226891 +0000 UTC m=+1425.902768644" watchObservedRunningTime="2026-11-03 07:22:50.713544024 +0000 UTC m=+1425.906085780" Nov 03 07:22:50.855800 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:50.851300 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" podStartSLOduration=2.851277082 podStartE2EDuration="2.851277082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:50.844444324 +0000 UTC m=+1426.036986078" watchObservedRunningTime="2026-11-03 07:22:50.851277082 +0000 UTC m=+1426.043818834" Nov 03 07:22:53.686860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.686825 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq"] Nov 03 07:22:53.690678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.690658 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.731352 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.731330 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4wjt\" (UniqueName: \"kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-kube-api-access-r4wjt\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.731455 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.731388 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.731510 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.731457 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.731510 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.731493 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.731603 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.731535 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.731666 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.731653 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.731704 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.731677 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.731777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.731706 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.731820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.731779 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.735369 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.735351 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc"] Nov 03 07:22:53.739561 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.739546 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:53.774738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.767217 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q"] Nov 03 07:22:53.847733 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.836228 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:53.847733 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.846009 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq"] Nov 03 07:22:53.847733 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.846169 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:53.862056 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.861957 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz"] Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.937439 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4wjt\" (UniqueName: \"kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-kube-api-access-r4wjt\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.937489 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.937589 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.937613 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.937650 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.937682 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.937705 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.937799 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmhg\" (UniqueName: \"kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-kube-api-access-chmhg\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.937838 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.937955 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.937987 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.938019 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd5682e-484f-4582-b4aa-261a78c77d75-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.938057 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.938086 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.938134 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.938167 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.938222 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.939149 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.939898 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istiod-ca-cert\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.940166 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-certs\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.940695 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-credential-socket\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.947285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.941225 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-data\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:53.949357 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.949335 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:53.999735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:53.985355 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-envoy\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:54.007689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.007659 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-podinfo\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:54.033349 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.027873 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc"] Nov 03 07:22:54.082784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.039624 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.083062 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.083032 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.144170 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.144233 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd5682e-484f-4582-b4aa-261a78c77d75-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.144318 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2dc7\" (UniqueName: \"kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-kube-api-access-s2dc7\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.144369 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.144410 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.144439 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.144533 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.144640 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.144673 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.144767 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmhg\" (UniqueName: \"kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-kube-api-access-chmhg\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.144868 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.144947 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.145002 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.145042 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.145105 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.145170 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.145753 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-certs\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.075111 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4wjt\" (UniqueName: \"kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-kube-api-access-r4wjt\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:54.149446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.130517 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q"] Nov 03 07:22:54.155336 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.153625 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.160348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.158975 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-data\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.160348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.160271 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-envoy\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.163546 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.163489 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-credential-socket\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.182444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.182413 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-token\") pod \"e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:54.187929 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.187904 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd5682e-484f-4582-b4aa-261a78c77d75-istiod-ca-cert\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.188125 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.188110 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz"] Nov 03 07:22:54.194763 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.194706 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2"] Nov 03 07:22:54.195252 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.195227 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-podinfo\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.200337 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.200274 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2"] Nov 03 07:22:54.200433 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.200381 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" Nov 03 07:22:54.215178 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.215152 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-token\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.232490 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.225425 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmhg\" (UniqueName: \"kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-kube-api-access-chmhg\") pod \"e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.384738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.381587 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-67db4bc0-dockercfg-ts2dd\"" Nov 03 07:22:54.384738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.381730 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-67db4bc0\"" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.404199 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.404259 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.404312 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.404532 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2dc7\" (UniqueName: \"kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-kube-api-access-s2dc7\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.404583 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.404625 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.404839 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.404887 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.404953 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.405166 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vm2\" (UniqueName: \"kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-kube-api-access-24vm2\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.405227 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.405267 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.405412 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.405606 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bfa7991-f674-43a6-82f0-174db2ff77fe-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.405646 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.405690 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38eead0-09aa-49ef-af09-1fafd54aec6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2\" (UID: \"c38eead0-09aa-49ef-af09-1fafd54aec6b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.405987 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.406051 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86cd\" (UniqueName: \"kubernetes.io/projected/c38eead0-09aa-49ef-af09-1fafd54aec6b-kube-api-access-d86cd\") pod \"payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2\" (UID: \"c38eead0-09aa-49ef-af09-1fafd54aec6b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.406125 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.407407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.406364 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.433016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.408447 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.433016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.408774 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.433016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.410208 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.433016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.411453 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.433016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.413141 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.433016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.413830 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.433016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.414268 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:54.433016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.417946 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:54.439753 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.439446 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.472525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.472483 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.473195 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.473160 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2dc7\" (UniqueName: \"kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-kube-api-access-s2dc7\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.508504 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.508475 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vm2\" (UniqueName: \"kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-kube-api-access-24vm2\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.508639 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.508536 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.508742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.508646 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bfa7991-f674-43a6-82f0-174db2ff77fe-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.508742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.508685 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38eead0-09aa-49ef-af09-1fafd54aec6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2\" (UID: \"c38eead0-09aa-49ef-af09-1fafd54aec6b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" Nov 03 07:22:54.508866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.508749 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.508866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.508795 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86cd\" (UniqueName: \"kubernetes.io/projected/c38eead0-09aa-49ef-af09-1fafd54aec6b-kube-api-access-d86cd\") pod \"payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2\" (UID: \"c38eead0-09aa-49ef-af09-1fafd54aec6b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" Nov 03 07:22:54.508866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.508850 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.509020 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.508893 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.509020 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.508933 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.509020 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.508997 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.509020 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.509020 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.509480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.509451 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.509763 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.509705 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-credential-socket\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.509872 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.509794 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38eead0-09aa-49ef-af09-1fafd54aec6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2\" (UID: \"c38eead0-09aa-49ef-af09-1fafd54aec6b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" Nov 03 07:22:54.511268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.511201 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-data\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.511472 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.511442 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-certs\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.512245 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.512201 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bfa7991-f674-43a6-82f0-174db2ff77fe-istiod-ca-cert\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.515534 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.515480 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-podinfo\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.515961 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.515900 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-envoy\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.549477 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.549300 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:54.577394 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.577367 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86cd\" (UniqueName: \"kubernetes.io/projected/c38eead0-09aa-49ef-af09-1fafd54aec6b-kube-api-access-d86cd\") pod \"payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2\" (UID: \"c38eead0-09aa-49ef-af09-1fafd54aec6b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" Nov 03 07:22:54.577394 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.577382 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vm2\" (UniqueName: \"kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-kube-api-access-24vm2\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.577873 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.577852 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-token\") pod \"e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.597843 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.597806 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:54.758833 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.758758 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" Nov 03 07:22:54.837071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.837048 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq"] Nov 03 07:22:54.838180 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:22:54.838155 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb42d5d_279b_4616_9b25_6d02bce7c1e3.slice/crio-a71dd0cc06090aeab48aa7f530277ce8951e59791c8faa7a7ac36d9f5e3af0ad WatchSource:0}: Error finding container a71dd0cc06090aeab48aa7f530277ce8951e59791c8faa7a7ac36d9f5e3af0ad: Status 404 returned error can't find the container with id a71dd0cc06090aeab48aa7f530277ce8951e59791c8faa7a7ac36d9f5e3af0ad Nov 03 07:22:54.841474 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.841443 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:54.841575 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.841522 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:54.841634 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.841573 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:54.882276 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.882248 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc"] Nov 03 07:22:54.889994 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:22:54.889937 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd5682e_484f_4582_b4aa_261a78c77d75.slice/crio-efd600313dda4733e7da34643957c7657d77fe366109850b611e258a59bcc004 WatchSource:0}: Error finding container efd600313dda4733e7da34643957c7657d77fe366109850b611e258a59bcc004: Status 404 returned error can't find the container with id efd600313dda4733e7da34643957c7657d77fe366109850b611e258a59bcc004 Nov 03 07:22:54.892989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.892953 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:54.893228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.893037 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:54.893228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.893080 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:54.947912 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.947206 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q"] Nov 03 07:22:54.950139 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.950106 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:54.950228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.950172 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:54.950228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:54.950201 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:55.008652 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.008617 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz"] Nov 03 07:22:55.024702 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.024616 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:55.025566 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.024901 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:55.026104 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.026078 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:22:55.167112 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.165118 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2"] Nov 03 07:22:55.651707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.651671 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" event={"ID":"4eb42d5d-279b-4616-9b25-6d02bce7c1e3","Type":"ContainerStarted","Data":"eb9f883d4804e759c4261fa38c4576097b598f43f1b4150abc0b997620564573"} Nov 03 07:22:55.651908 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.651766 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" event={"ID":"4eb42d5d-279b-4616-9b25-6d02bce7c1e3","Type":"ContainerStarted","Data":"a71dd0cc06090aeab48aa7f530277ce8951e59791c8faa7a7ac36d9f5e3af0ad"} Nov 03 07:22:55.654979 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.654955 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" event={"ID":"c38eead0-09aa-49ef-af09-1fafd54aec6b","Type":"ContainerStarted","Data":"e40a44001bea0e34d2c9903f28f70dc38270f6081f100ecdd859f30b03b9bd8a"} Nov 03 07:22:55.655129 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.655117 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" event={"ID":"c38eead0-09aa-49ef-af09-1fafd54aec6b","Type":"ContainerStarted","Data":"f075b685f067031de40eb11bab112570a2128e0725c84ed662f81b3219226683"} Nov 03 07:22:55.656614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.656592 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" event={"ID":"9bfa7991-f674-43a6-82f0-174db2ff77fe","Type":"ContainerStarted","Data":"db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0"} Nov 03 07:22:55.656729 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.656635 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" event={"ID":"9bfa7991-f674-43a6-82f0-174db2ff77fe","Type":"ContainerStarted","Data":"683edc5bfd59efe33a8bd1547c8a7d755a345bd5ee1c5bf00e9bbc2a4c0b0844"} Nov 03 07:22:55.658155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.658133 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" event={"ID":"9fd5682e-484f-4582-b4aa-261a78c77d75","Type":"ContainerStarted","Data":"44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6"} Nov 03 07:22:55.658248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.658181 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" event={"ID":"9fd5682e-484f-4582-b4aa-261a78c77d75","Type":"ContainerStarted","Data":"efd600313dda4733e7da34643957c7657d77fe366109850b611e258a59bcc004"} Nov 03 07:22:55.659518 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.659497 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" event={"ID":"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf","Type":"ContainerStarted","Data":"976e956f48846d56b062924f716f399ae726ff11611f026f0cf9c05772e2a4f6"} Nov 03 07:22:55.659588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.659542 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" event={"ID":"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf","Type":"ContainerStarted","Data":"d7acc8245d6b77b7c751a2b00a85ce15d02909c3fbd6432991708e7779af4da1"} Nov 03 07:22:55.932605 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.932497 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" podStartSLOduration=1.9324787159999999 podStartE2EDuration="1.932478716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:55.915981234 +0000 UTC m=+1431.108522986" watchObservedRunningTime="2026-11-03 07:22:55.932478716 +0000 UTC m=+1431.125020465" Nov 03 07:22:55.933523 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:55.933478 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podStartSLOduration=2.933467996 podStartE2EDuration="2.933467996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:55.78066509 +0000 UTC m=+1430.973206841" watchObservedRunningTime="2026-11-03 07:22:55.933467996 +0000 UTC m=+1431.126009741" Nov 03 07:22:56.209791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.207260 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podStartSLOduration=3.207237777 podStartE2EDuration="3.207237777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:56.054497303 +0000 UTC m=+1431.247039054" watchObservedRunningTime="2026-11-03 07:22:56.207237777 +0000 UTC m=+1431.399779529" Nov 03 07:22:56.357743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.353342 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podStartSLOduration=3.353319679 podStartE2EDuration="3.353319679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:56.209336254 +0000 UTC m=+1431.401878007" watchObservedRunningTime="2026-11-03 07:22:56.353319679 +0000 UTC m=+1431.545861430" Nov 03 07:22:56.396739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.393860 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podStartSLOduration=3.393837245 podStartE2EDuration="3.393837245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:22:56.341794125 +0000 UTC m=+1431.534335876" watchObservedRunningTime="2026-11-03 07:22:56.393837245 +0000 UTC m=+1431.586378996" Nov 03 07:22:56.442112 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.442079 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:22:56.452245 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.452220 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:22:56.452479 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.443598 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:22:56.452611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.452591 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 07:22:56.453002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.452136 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:22:56.453158 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.453135 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:22:56.562420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.562383 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:22:56.565076 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.564958 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:22:56.565305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.565271 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:22:56.608727 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.608680 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:22:56.610667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.610639 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:22:56.610951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.610923 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:22:56.765285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:56.765250 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" Nov 03 07:22:57.050180 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:57.050081 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:57.099869 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:57.099821 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:57.103747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:57.102520 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 07:22:57.387890 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:57.387849 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:57.431309 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:57.431274 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:22:57.431632 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:57.431578 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 07:22:57.436099 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:57.436066 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:22:57.436250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:57.436133 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:22:57.561569 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:57.561536 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:22:57.561854 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:57.561828 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:22:57.605858 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:57.605225 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:22:57.605858 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:57.605294 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:22:58.176739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.165634 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 07:22:58.176739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.166387 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:22:58.407378 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.407334 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:58.409178 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.409155 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:22:58.441791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.429968 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:22:58.441791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.430027 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 07:22:58.497739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.488489 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:22:58.497739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.488541 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:22:58.660739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.553041 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:22:58.660739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.553096 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:22:58.777970 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.777937 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:22:58.778216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.778192 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:22:58.922108 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:58.880406 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:59.180770 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.168964 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:59.262752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.258310 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:59.264280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.264249 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:22:59.264663 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.264645 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 07:22:59.341744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.339000 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:59.453105 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.436807 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:22:59.470012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.469973 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Nov 03 07:22:59.477279 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.475503 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:22:59.477279 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.475559 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 07:22:59.583876 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.583838 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:22:59.584140 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.584115 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:22:59.604772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.590217 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:22:59.604772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.590270 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:22:59.604772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.601957 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:22:59.604772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:22:59.602019 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:23:00.273737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.273063 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:00.273737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.273429 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:00.273737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.273674 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Nov 03 07:23:00.274384 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.273789 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:23:00.274384 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.273850 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:23:00.277735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.275693 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:00.376413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.376374 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:00.386876 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.386844 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:23:00.442085 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.442052 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:23:00.462837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.462792 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 07:23:00.463073 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.442306 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:00.467705 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.467589 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:00.467953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.467929 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:23:00.556176 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.553339 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:00.556176 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.553399 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:23:00.612738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.604497 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:00.612738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:00.604558 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:23:01.318786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.314334 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:01.351871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.351818 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 07:23:01.363461 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.363409 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:01.438762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.380814 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 07:23:01.440748 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.439168 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:01.440748 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.439226 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:23:01.542881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.500226 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:01.542881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.502945 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:23:01.542881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.503008 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 07:23:01.689825 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.674312 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:01.689825 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.674365 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:23:01.708806 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.701763 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:01.708806 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.701847 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:23:01.708806 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.706890 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" Nov 03 07:23:01.756738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.756021 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" Nov 03 07:23:01.917839 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:01.917791 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:02.265742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.233005 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:23:02.265742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.251068 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:23:02.319741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.301260 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:23:02.320762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.320301 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 07:23:02.382456 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.382413 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:02.425743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.417830 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:02.488739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.486431 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:23:02.488739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.486497 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 07:23:02.488739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.486617 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:02.488739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.486643 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:23:02.540902 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.540803 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 07:23:02.581740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.572235 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:02.582054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.582018 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:23:02.654738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.637209 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:02.658843 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.658794 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:23:02.779741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.767119 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:23:02.897748 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.869032 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:23:03.010781 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:02.992793 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 07:23:03.262821 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.258910 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 07:23:03.276953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.276913 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:23:03.302191 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.300479 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:23:03.304359 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.304336 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:23:03.383895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.383843 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:03.456230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.455530 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:03.456230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.455586 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:23:03.459985 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.459959 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:23:03.484038 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.483999 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 07:23:03.594794 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.556934 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:03.594794 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.556981 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:23:03.803352 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.802255 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 07:23:03.836880 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.822434 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:03.836880 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.822486 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:23:03.994899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:03.994792 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 07:23:04.224909 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.224845 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:04.279032 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.278943 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:04.280438 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.280413 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:23:04.280682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.280669 4409 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 07:23:04.322787 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.322526 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:04.374883 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.374842 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps"] Nov 03 07:23:04.401736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.397102 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.402360 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.402323 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:04.425704 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.425673 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:23:04.433745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.425955 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 07:23:04.442991 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.442866 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:04.442991 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.442919 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:23:04.472772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.472738 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-istio-token\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.511560 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.511522 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-certs\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.512262 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.512232 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f64254-ffd6-4900-94fa-c6386145579a-istiod-ca-cert\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.512502 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.512481 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-envoy\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.512752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.512732 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-data\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.512903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.512886 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-socket\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.513056 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.513041 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f64254-ffd6-4900-94fa-c6386145579a-istio-podinfo\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.513290 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.513270 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-credential-socket\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.513458 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.513442 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j7zb\" (UniqueName: \"kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-kube-api-access-7j7zb\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.514397 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.511400 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps"] Nov 03 07:23:04.533275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.533203 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5c169f6a-openshift-default-dockercfg-ppk98\"" Nov 03 07:23:04.561855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.561799 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:04.695739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.694610 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f64254-ffd6-4900-94fa-c6386145579a-istiod-ca-cert\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.695739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.694666 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-envoy\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.695739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.694678 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:04.695739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.694752 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:23:04.712218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.704825 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f64254-ffd6-4900-94fa-c6386145579a-istiod-ca-cert\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.712414 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.712318 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:04.712414 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.712369 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:23:04.730756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.720922 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-data\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.730756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.720983 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-socket\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.730756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.721053 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f64254-ffd6-4900-94fa-c6386145579a-istio-podinfo\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.730756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.721195 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-credential-socket\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.730756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.721255 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j7zb\" (UniqueName: \"kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-kube-api-access-7j7zb\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.730756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.721369 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-istio-token\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.730756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.721428 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-certs\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.730756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.721952 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-certs\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.730756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.722231 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-credential-socket\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.730756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.722526 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-data\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.730756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.723108 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-socket\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:04.813789 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.810811 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:04.906860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:04.905426 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:23:05.224726 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.212950 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j7zb\" (UniqueName: \"kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-kube-api-access-7j7zb\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:05.267743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.239080 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-envoy\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:05.267743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.262568 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 07:23:05.342791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.332954 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f64254-ffd6-4900-94fa-c6386145579a-istio-podinfo\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:05.371017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.352805 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.371017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.355102 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 07:23:05.474020 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.473971 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:05.474654 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.474617 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:23:05.484742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.479206 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.484742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.480647 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Nov 03 07:23:05.611773 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.500124 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 07:23:05.611773 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.501220 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:23:05.611773 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.501266 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 07:23:05.611773 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.502562 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-istio-token\") pod \"e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:05.704760 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.704703 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:05.920995 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.920953 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:23:05.921268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.892496 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 07:23:05.921380 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.715895 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:05.921503 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.714423 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:05.921629 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:05.921604 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:23:06.038641 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.038604 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5c169f6a-openshift-default-dockercfg-ppk98\"" Nov 03 07:23:06.063764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.061191 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:06.245783 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:23:06.245350 4409 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.338s" Nov 03 07:23:06.247544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.247505 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:23:06.247910 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.247891 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:23:06.282181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.282082 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.387937 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.368692 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:23:06.387937 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.368880 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 07:23:06.458739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.455859 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.462736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.459618 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 07:23:06.512511 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.512471 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 07:23:06.513046 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.486589 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:06.590442 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.489911 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:06.591489 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.591455 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:23:06.591848 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.590685 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:06.592000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.591980 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:23:06.650063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.649989 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 07:23:06.698741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.683112 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:06.698741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:06.683170 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:23:07.274910 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.274854 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 07:23:07.283955 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.283916 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.369753 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.288891 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.371735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.371681 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.418476 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.375791 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:23:07.478772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.460842 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:07.478772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.460900 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:23:07.513852 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.513807 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:07.514422 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.514392 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.564809 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.559497 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:07.564809 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.559549 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:23:07.615841 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.615792 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.666742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.649922 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:07.698868 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.698824 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:23:07.699119 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.674016 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.724741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.672682 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:23:07.946895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.946796 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:07.947165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.947136 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.965209 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.965175 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:07.993703 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:07.993673 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" Nov 03 07:23:08.295792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.294885 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.406559 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.406511 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 07:23:08.446579 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.417564 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 07:23:08.459069 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.444219 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:23:08.501375 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.444706 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:08.526740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.499945 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:08.527986 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.527569 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.579512 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.502218 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:23:08.615742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.607133 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 07:23:08.615742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.607194 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 07:23:08.615742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.608969 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:08.615742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.609011 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:23:08.691740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.690357 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 07:23:08.766747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.707764 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.766747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.708405 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 07:23:08.766747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.751805 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:08.766747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.751862 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:08.766747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.752829 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:08.766747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:08.752875 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.349171 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.349128 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.441738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.435198 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 07:23:09.464759 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.458831 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.489941 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.488414 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 07:23:09.489941 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.488480 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 07:23:09.490558 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.490527 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 07:23:09.508766 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.508727 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:23:09.622303 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.618100 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:09.622303 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.618154 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.662889 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.662853 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps"] Nov 03 07:23:09.663171 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.663158 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:23:09.663320 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.663295 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:23:09.757256 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.675227 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:23:09.757610 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.757593 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:23:09.758032 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.758018 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:23:09.780081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.726369 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 07:23:09.780081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.768816 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 07:23:09.780081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.698784 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.780081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.754965 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.782528 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:23:09.780606 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:23:09.782528 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.720804 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:09.800206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.791092 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:23:09.838944 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.838440 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:23:09.838944 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.838536 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:23:09.838944 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:09.838587 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:23:10.370851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:10.370808 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:10.403328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:10.403292 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 07:23:10.733756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:10.436291 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:10.737054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:10.559371 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:10.737054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:10.681928 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:10.809390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:10.779449 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 07:23:10.809390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:10.784491 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:10.811890 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:10.810426 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:10.835751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:10.817006 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:10.836474 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:10.836440 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:10.856651 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:10.856611 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" event={"ID":"80f64254-ffd6-4900-94fa-c6386145579a","Type":"ContainerStarted","Data":"23107a75228928b955adc18ec37e2c9e1ea580364f6494ad858fbfcde0980043"} Nov 03 07:23:11.636890 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.635456 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:23:11.636890 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.635522 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:23:11.636890 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.635563 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:23:11.764975 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.764935 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:11.768809 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.768773 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:23:11.783313 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.768933 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:11.783313 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.779776 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 03 07:23:11.783313 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.780937 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:23:11.796836 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.796801 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:11.851437 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.851392 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:11.880345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.805234 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:11.880345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.853479 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:11.880345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.855213 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:23:11.929473 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.925932 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:23:11.930002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.929972 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:23:11.934198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.934172 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" event={"ID":"80f64254-ffd6-4900-94fa-c6386145579a","Type":"ContainerStarted","Data":"8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b"} Nov 03 07:23:11.941050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.941030 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:23:11.966372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.966338 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:11.980943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:11.980908 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:12.113226 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:12.111097 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:12.114777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:12.114747 4409 patch_prober.go:29] interesting pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:23:12.115009 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:12.114975 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 07:23:12.481995 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:12.481966 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:23:12.829737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:12.820611 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:12.904834 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:12.904798 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:12.905103 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:12.905074 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:12.937739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:12.907954 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:23:13.052738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:13.043849 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:13.052738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:13.043913 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:13.087081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:13.087005 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:13.293396 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:13.285102 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:23:13.299266 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:13.299240 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:23:13.363416 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:13.363295 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" podStartSLOduration=9.363278687 podStartE2EDuration="9.363278687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:13.334021038 +0000 UTC m=+1448.526562788" watchObservedRunningTime="2026-11-03 07:23:13.363278687 +0000 UTC m=+1448.555820437" Nov 03 07:23:13.978008 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:13.977960 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:13.978508 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:13.978024 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:14.042397 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:14.042366 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:14.044517 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:14.044484 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:23:14.070992 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:14.070964 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:23:14.103578 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:14.103555 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:23:20.232739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.229206 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t"] Nov 03 07:23:20.261034 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.261002 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t"] Nov 03 07:23:20.261505 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.261425 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" Nov 03 07:23:20.402755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.399703 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqf2h\" (UniqueName: \"kubernetes.io/projected/2d42febe-a8a4-4b3c-b449-e19b9c957409-kube-api-access-fqf2h\") pod \"payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t\" (UID: \"2d42febe-a8a4-4b3c-b449-e19b9c957409\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" Nov 03 07:23:20.402755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.399792 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d42febe-a8a4-4b3c-b449-e19b9c957409-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t\" (UID: \"2d42febe-a8a4-4b3c-b449-e19b9c957409\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" Nov 03 07:23:20.408348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.408320 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5c169f6a-dockercfg-rld92\"" Nov 03 07:23:20.476579 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.473981 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5c169f6a\"" Nov 03 07:23:20.531859 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.514941 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqf2h\" (UniqueName: \"kubernetes.io/projected/2d42febe-a8a4-4b3c-b449-e19b9c957409-kube-api-access-fqf2h\") pod \"payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t\" (UID: \"2d42febe-a8a4-4b3c-b449-e19b9c957409\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" Nov 03 07:23:20.531859 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.515056 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d42febe-a8a4-4b3c-b449-e19b9c957409-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t\" (UID: \"2d42febe-a8a4-4b3c-b449-e19b9c957409\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" Nov 03 07:23:20.531859 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.516405 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d42febe-a8a4-4b3c-b449-e19b9c957409-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t\" (UID: \"2d42febe-a8a4-4b3c-b449-e19b9c957409\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" Nov 03 07:23:20.600471 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.600434 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8"] Nov 03 07:23:20.614558 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.614513 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqf2h\" (UniqueName: \"kubernetes.io/projected/2d42febe-a8a4-4b3c-b449-e19b9c957409-kube-api-access-fqf2h\") pod \"payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t\" (UID: \"2d42febe-a8a4-4b3c-b449-e19b9c957409\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" Nov 03 07:23:20.623848 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.623822 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8"] Nov 03 07:23:20.624055 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.623921 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" Nov 03 07:23:20.719358 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.719323 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9bkr\" (UniqueName: \"kubernetes.io/projected/027e5dc0-f742-4388-a8dd-62de2f7651d2-kube-api-access-w9bkr\") pod \"payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8\" (UID: \"027e5dc0-f742-4388-a8dd-62de2f7651d2\") " pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" Nov 03 07:23:20.719559 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.719532 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027e5dc0-f742-4388-a8dd-62de2f7651d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8\" (UID: \"027e5dc0-f742-4388-a8dd-62de2f7651d2\") " pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" Nov 03 07:23:20.738016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.737989 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" Nov 03 07:23:20.821898 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.821386 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027e5dc0-f742-4388-a8dd-62de2f7651d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8\" (UID: \"027e5dc0-f742-4388-a8dd-62de2f7651d2\") " pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" Nov 03 07:23:20.821898 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.821646 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9bkr\" (UniqueName: \"kubernetes.io/projected/027e5dc0-f742-4388-a8dd-62de2f7651d2-kube-api-access-w9bkr\") pod \"payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8\" (UID: \"027e5dc0-f742-4388-a8dd-62de2f7651d2\") " pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" Nov 03 07:23:20.823756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.823150 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027e5dc0-f742-4388-a8dd-62de2f7651d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8\" (UID: \"027e5dc0-f742-4388-a8dd-62de2f7651d2\") " pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" Nov 03 07:23:20.887143 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.887115 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9bkr\" (UniqueName: \"kubernetes.io/projected/027e5dc0-f742-4388-a8dd-62de2f7651d2-kube-api-access-w9bkr\") pod \"payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8\" (UID: \"027e5dc0-f742-4388-a8dd-62de2f7651d2\") " pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" Nov 03 07:23:20.944805 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:20.944773 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" Nov 03 07:23:21.133628 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:21.133599 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t"] Nov 03 07:23:21.133879 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:23:21.133858 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d42febe_a8a4_4b3c_b449_e19b9c957409.slice/crio-9b2e8ae669b5f8caf668adc604579c34b93404589aed1888083d896a0bb53914 WatchSource:0}: Error finding container 9b2e8ae669b5f8caf668adc604579c34b93404589aed1888083d896a0bb53914: Status 404 returned error can't find the container with id 9b2e8ae669b5f8caf668adc604579c34b93404589aed1888083d896a0bb53914 Nov 03 07:23:21.329377 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:21.329356 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8"] Nov 03 07:23:21.329883 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:23:21.329857 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027e5dc0_f742_4388_a8dd_62de2f7651d2.slice/crio-a19f9cd6dc190351d85a514fe54b77b47c7500b4b17dd4acb6d7280c254a0968 WatchSource:0}: Error finding container a19f9cd6dc190351d85a514fe54b77b47c7500b4b17dd4acb6d7280c254a0968: Status 404 returned error can't find the container with id a19f9cd6dc190351d85a514fe54b77b47c7500b4b17dd4acb6d7280c254a0968 Nov 03 07:23:22.127905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:22.127411 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" event={"ID":"2d42febe-a8a4-4b3c-b449-e19b9c957409","Type":"ContainerStarted","Data":"58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6"} Nov 03 07:23:22.128264 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:22.128232 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" Nov 03 07:23:22.128398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:22.128277 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" event={"ID":"2d42febe-a8a4-4b3c-b449-e19b9c957409","Type":"ContainerStarted","Data":"9b2e8ae669b5f8caf668adc604579c34b93404589aed1888083d896a0bb53914"} Nov 03 07:23:22.129593 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:22.129571 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" event={"ID":"027e5dc0-f742-4388-a8dd-62de2f7651d2","Type":"ContainerStarted","Data":"bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e"} Nov 03 07:23:22.129695 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:22.129609 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" event={"ID":"027e5dc0-f742-4388-a8dd-62de2f7651d2","Type":"ContainerStarted","Data":"a19f9cd6dc190351d85a514fe54b77b47c7500b4b17dd4acb6d7280c254a0968"} Nov 03 07:23:22.129695 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:22.129650 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" Nov 03 07:23:22.264750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:22.259902 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" podStartSLOduration=2.259881458 podStartE2EDuration="2.259881458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:22.245914097 +0000 UTC m=+1457.438455850" watchObservedRunningTime="2026-11-03 07:23:22.259881458 +0000 UTC m=+1457.452423210" Nov 03 07:23:22.395787 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:22.393202 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" podStartSLOduration=2.393179179 podStartE2EDuration="2.393179179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:22.375521145 +0000 UTC m=+1457.568062890" watchObservedRunningTime="2026-11-03 07:23:22.393179179 +0000 UTC m=+1457.585720929" Nov 03 07:23:32.448616 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.448582 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt"] Nov 03 07:23:32.454514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.454493 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:32.505879 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.505855 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-metrics-certs\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:32.505989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.505925 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-maas-api-tls\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:32.506039 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.506020 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvzfj\" (UniqueName: \"kubernetes.io/projected/29b881ff-f8e3-4af0-9f26-2fbe5720825b-kube-api-access-cvzfj\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:32.506161 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.506145 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b881ff-f8e3-4af0-9f26-2fbe5720825b-openshift-service-ca\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:32.557641 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.557603 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-67db4bc0-serving-cert\"" Nov 03 07:23:32.561542 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.561520 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt"] Nov 03 07:23:32.607887 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.607859 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b881ff-f8e3-4af0-9f26-2fbe5720825b-openshift-service-ca\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:32.608064 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.608046 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-metrics-certs\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:32.608185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.608168 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-maas-api-tls\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:32.608242 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.608227 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvzfj\" (UniqueName: \"kubernetes.io/projected/29b881ff-f8e3-4af0-9f26-2fbe5720825b-kube-api-access-cvzfj\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:32.608336 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:23:32.608314 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-serving-cert: secret "maas-api-e2e-ait-67db4bc0-serving-cert" not found Nov 03 07:23:32.608412 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:23:32.608398 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-maas-api-tls podName:29b881ff-f8e3-4af0-9f26-2fbe5720825b nodeName:}" failed. No retries permitted until 2026-11-03 07:23:33.108379377 +0000 UTC m=+1468.300921122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-maas-api-tls") pod "maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" (UID: "29b881ff-f8e3-4af0-9f26-2fbe5720825b") : secret "maas-api-e2e-ait-67db4bc0-serving-cert" not found Nov 03 07:23:32.608600 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.608577 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b881ff-f8e3-4af0-9f26-2fbe5720825b-openshift-service-ca\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:32.610680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.610663 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-metrics-certs\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:32.670902 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:32.670883 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvzfj\" (UniqueName: \"kubernetes.io/projected/29b881ff-f8e3-4af0-9f26-2fbe5720825b-kube-api-access-cvzfj\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:33.140264 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.140236 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" Nov 03 07:23:33.148019 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.147984 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-maas-api-tls\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:33.150389 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.150369 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-maas-api-tls\") pod \"maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:33.369331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.369298 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:33.519478 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.519443 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g"] Nov 03 07:23:33.525530 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.525497 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.587569 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.587517 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-89ggc\"" Nov 03 07:23:33.591650 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.591628 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g"] Nov 03 07:23:33.657249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.657172 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.657417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.657274 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.657486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.657422 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.657486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.657473 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620abc13-9e8b-440f-a23a-f230d0656275-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.657634 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.657505 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.657634 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.657569 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx9nz\" (UniqueName: \"kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-kube-api-access-nx9nz\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.657634 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.657604 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.657821 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.657677 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620abc13-9e8b-440f-a23a-f230d0656275-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.657911 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.657891 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.759336 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.759304 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.759518 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.759355 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.759518 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.759400 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.759518 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.759463 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.759518 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.759492 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620abc13-9e8b-440f-a23a-f230d0656275-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.759807 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.759628 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.759807 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.759745 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx9nz\" (UniqueName: \"kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-kube-api-access-nx9nz\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.759807 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.759786 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.759970 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.759849 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.759970 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.759861 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620abc13-9e8b-440f-a23a-f230d0656275-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.759970 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.759874 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.760129 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.760109 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.760261 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.760242 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620abc13-9e8b-440f-a23a-f230d0656275-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.760320 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.760268 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.761873 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.761856 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.762548 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.762531 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620abc13-9e8b-440f-a23a-f230d0656275-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.792931 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.792911 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt"] Nov 03 07:23:33.793318 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:23:33.793298 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b881ff_f8e3_4af0_9f26_2fbe5720825b.slice/crio-28cc65389cc3a966f3b72bb3e63b2d234de434834d77c5f732ffdb68b36037ad WatchSource:0}: Error finding container 28cc65389cc3a966f3b72bb3e63b2d234de434834d77c5f732ffdb68b36037ad: Status 404 returned error can't find the container with id 28cc65389cc3a966f3b72bb3e63b2d234de434834d77c5f732ffdb68b36037ad Nov 03 07:23:33.795254 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.795236 4409 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:23:33.822482 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.822453 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.822582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.822515 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx9nz\" (UniqueName: \"kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-kube-api-access-nx9nz\") pod \"maas-default-gateway-openshift-default-58d898bd55-s4f8g\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:33.844245 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:33.844224 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:34.147402 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:34.147375 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" Nov 03 07:23:34.207999 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:34.207972 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" event={"ID":"29b881ff-f8e3-4af0-9f26-2fbe5720825b","Type":"ContainerStarted","Data":"28cc65389cc3a966f3b72bb3e63b2d234de434834d77c5f732ffdb68b36037ad"} Nov 03 07:23:34.280343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:34.280310 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g"] Nov 03 07:23:34.280479 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:23:34.280423 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620abc13_9e8b_440f_a23a_f230d0656275.slice/crio-30071f8415a2173d39a565ee169cbfc4a3f2de75d13d6ee6a8e95bfebfa3492d WatchSource:0}: Error finding container 30071f8415a2173d39a565ee169cbfc4a3f2de75d13d6ee6a8e95bfebfa3492d: Status 404 returned error can't find the container with id 30071f8415a2173d39a565ee169cbfc4a3f2de75d13d6ee6a8e95bfebfa3492d Nov 03 07:23:34.283290 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:34.283255 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:23:34.283403 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:34.283334 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:23:34.283403 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:34.283378 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:23:35.226222 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:35.225731 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" event={"ID":"29b881ff-f8e3-4af0-9f26-2fbe5720825b","Type":"ContainerStarted","Data":"13251e72d3091799b018efd7c69081797debd5736af0070058e78047834af9b6"} Nov 03 07:23:35.226222 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:35.225849 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:35.227944 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:35.227918 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" event={"ID":"620abc13-9e8b-440f-a23a-f230d0656275","Type":"ContainerStarted","Data":"308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82"} Nov 03 07:23:35.228083 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:35.227964 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" event={"ID":"620abc13-9e8b-440f-a23a-f230d0656275","Type":"ContainerStarted","Data":"30071f8415a2173d39a565ee169cbfc4a3f2de75d13d6ee6a8e95bfebfa3492d"} Nov 03 07:23:35.349542 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:35.349480 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" podStartSLOduration=2.530540685 podStartE2EDuration="3.349463411s" totalImagesPullingTime="818.922726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:35.348360574 +0000 UTC m=+1470.540902347" watchObservedRunningTime="2026-11-03 07:23:35.349463411 +0000 UTC m=+1470.542005162" Nov 03 07:23:35.853779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:35.853687 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:35.854161 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:35.854070 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:23:35.854352 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:35.854324 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 07:23:36.845273 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:36.845237 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:23:36.845668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:36.845292 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 07:23:37.742265 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.742213 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" podStartSLOduration=4.742197737 podStartE2EDuration="4.742197737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:35.501156965 +0000 UTC m=+1470.693698717" watchObservedRunningTime="2026-11-03 07:23:37.742197737 +0000 UTC m=+1472.934739487" Nov 03 07:23:37.743863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.743839 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv"] Nov 03 07:23:37.748822 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.748802 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:37.805049 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.805012 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljhw\" (UniqueName: \"kubernetes.io/projected/d40c18a0-45b8-4612-a17d-b18c0599623b-kube-api-access-8ljhw\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:37.805512 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.805485 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-maas-api-tls\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:37.805622 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.805542 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d40c18a0-45b8-4612-a17d-b18c0599623b-openshift-service-ca\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:37.805697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.805661 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-metrics-certs\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:37.809513 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.809489 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv"] Nov 03 07:23:37.844632 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.844605 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:23:37.844787 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.844661 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 07:23:37.854896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.854874 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-33a46b37-serving-cert\"" Nov 03 07:23:37.907069 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.907028 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-metrics-certs\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:37.907250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.907118 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljhw\" (UniqueName: \"kubernetes.io/projected/d40c18a0-45b8-4612-a17d-b18c0599623b-kube-api-access-8ljhw\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:37.907385 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.907365 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-maas-api-tls\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:37.907454 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.907415 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d40c18a0-45b8-4612-a17d-b18c0599623b-openshift-service-ca\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:37.907788 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:23:37.907760 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-serving-cert: secret "maas-api-e2e-migrate-33a46b37-serving-cert" not found Nov 03 07:23:37.907912 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:23:37.907877 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-maas-api-tls podName:d40c18a0-45b8-4612-a17d-b18c0599623b nodeName:}" failed. No retries permitted until 2026-11-03 07:23:38.407851365 +0000 UTC m=+1473.600393114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-maas-api-tls") pod "maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" (UID: "d40c18a0-45b8-4612-a17d-b18c0599623b") : secret "maas-api-e2e-migrate-33a46b37-serving-cert" not found Nov 03 07:23:37.908332 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.908300 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d40c18a0-45b8-4612-a17d-b18c0599623b-openshift-service-ca\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:37.910570 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.910543 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-metrics-certs\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:37.991500 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:37.991468 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljhw\" (UniqueName: \"kubernetes.io/projected/d40c18a0-45b8-4612-a17d-b18c0599623b-kube-api-access-8ljhw\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:38.460185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:38.460149 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-maas-api-tls\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:38.462767 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:38.462745 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-maas-api-tls\") pod \"maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:38.663176 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:38.663141 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:38.808683 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:23:38.808644 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:23:38.844987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:38.844916 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:23:38.844987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:38.844976 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 07:23:39.157738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.154815 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv"] Nov 03 07:23:39.220030 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:23:39.219987 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40c18a0_45b8_4612_a17d_b18c0599623b.slice/crio-efddfb282f18b46ebb601c04acdab6372b0cd640042ab90a63113269c40bfa1b WatchSource:0}: Error finding container efddfb282f18b46ebb601c04acdab6372b0cd640042ab90a63113269c40bfa1b: Status 404 returned error can't find the container with id efddfb282f18b46ebb601c04acdab6372b0cd640042ab90a63113269c40bfa1b Nov 03 07:23:39.259738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.255118 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw"] Nov 03 07:23:39.279761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.279580 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.372263 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.368387 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" event={"ID":"d40c18a0-45b8-4612-a17d-b18c0599623b","Type":"ContainerStarted","Data":"efddfb282f18b46ebb601c04acdab6372b0cd640042ab90a63113269c40bfa1b"} Nov 03 07:23:39.402737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.402075 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw"] Nov 03 07:23:39.457854 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.428035 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.457854 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.428160 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.457854 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.428195 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.457854 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.428384 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.457854 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.428482 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.457854 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.428581 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.457854 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.428757 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.457854 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.428829 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbfmt\" (UniqueName: \"kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-kube-api-access-tbfmt\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.457854 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.430957 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.532472 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.532431 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.532662 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.532524 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.532662 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.532544 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.532662 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.532561 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.532662 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.532614 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.532662 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.532642 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.532851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.532679 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.532851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.532743 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.532851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.532768 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbfmt\" (UniqueName: \"kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-kube-api-access-tbfmt\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.533351 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.533312 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.533629 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.533609 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-certs\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.533821 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.533793 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-credential-socket\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.533957 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.533936 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istiod-ca-cert\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.534034 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.533972 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-data\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.537051 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.537027 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-envoy\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.537291 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.537271 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-podinfo\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.596022 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.595986 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-token\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.597736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.597071 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbfmt\" (UniqueName: \"kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-kube-api-access-tbfmt\") pod \"e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.688065 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.688026 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:39.847351 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.847303 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:23:39.847561 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:39.847374 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 07:23:40.164132 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.164095 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw"] Nov 03 07:23:40.195979 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:23:40.188330 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3c1f7d_b9c6_4725_a926_1eb85ecaf7ed.slice/crio-1b2a72e67eeae817b7c2401162348366d5c72a90da883ef18d9242508bba8532 WatchSource:0}: Error finding container 1b2a72e67eeae817b7c2401162348366d5c72a90da883ef18d9242508bba8532: Status 404 returned error can't find the container with id 1b2a72e67eeae817b7c2401162348366d5c72a90da883ef18d9242508bba8532 Nov 03 07:23:40.195979 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.194238 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:23:40.195979 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.194329 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:23:40.195979 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.194372 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:23:40.409066 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.409027 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" event={"ID":"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed","Type":"ContainerStarted","Data":"87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e"} Nov 03 07:23:40.409265 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.409101 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" event={"ID":"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed","Type":"ContainerStarted","Data":"1b2a72e67eeae817b7c2401162348366d5c72a90da883ef18d9242508bba8532"} Nov 03 07:23:40.411306 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.411282 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" event={"ID":"d40c18a0-45b8-4612-a17d-b18c0599623b","Type":"ContainerStarted","Data":"cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed"} Nov 03 07:23:40.411515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.411494 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:40.533358 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.533243 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" podStartSLOduration=1.533221994 podStartE2EDuration="1.533221994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:40.531995954 +0000 UTC m=+1475.724537705" watchObservedRunningTime="2026-11-03 07:23:40.533221994 +0000 UTC m=+1475.725763744" Nov 03 07:23:40.666283 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.666211 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" podStartSLOduration=2.688935094 podStartE2EDuration="3.666190481s" totalImagesPullingTime="977.255387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:23:40.665347208 +0000 UTC m=+1475.857888959" watchObservedRunningTime="2026-11-03 07:23:40.666190481 +0000 UTC m=+1475.858732232" Nov 03 07:23:40.689662 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.689630 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:40.849177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.849108 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:23:40.849376 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:40.849177 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 07:23:41.304031 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:41.301927 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:23:41.695798 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:41.695182 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 07:23:41.695798 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:41.695270 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" podUID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 07:23:41.845406 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:41.845371 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 07:23:41.845589 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:41.845425 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 07:23:42.691348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:42.691302 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 07:23:42.691926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:42.691390 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" podUID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 07:23:43.692740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:43.692091 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 07:23:43.692740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:43.692155 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" podUID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 07:23:43.788739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:43.784061 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:43.788739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:43.788062 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:43.795976 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:43.795956 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:23:44.770737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:44.695276 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 07:23:44.794664 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:44.794625 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" podUID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 07:23:45.690855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:45.690817 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 07:23:45.691009 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:45.690877 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" podUID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 07:23:46.423796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:46.423687 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:23:46.690400 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:46.690300 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 07:23:46.690665 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:46.690623 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" podUID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 07:23:48.148444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:48.148409 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:48.148967 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:48.148855 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:48.150121 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:48.150098 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:23:57.746358 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:57.746271 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:58.388738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:58.384802 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 03 07:23:58.754919 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:58.754858 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:58.755882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:58.755857 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:23:58.802736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:58.802294 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:58.958740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:58.945416 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:59.027889 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:59.027786 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 07:23:59.378796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:59.373374 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 07:23:59.528738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:59.390895 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:59.559017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:59.558604 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:59.559017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:59.558676 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:59.595241 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:59.595197 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:23:59.732334 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:59.732141 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:23:59.855741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:59.844219 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:23:59.855741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:23:59.844281 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:24:05.103178 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:05.103152 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:24:05.103631 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:05.103153 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:24:07.882830 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:07.882800 4409 generic.go:356] "Generic (PLEG): container finished" podID="94f95955-229f-446c-ac9f-2fab451a3889" containerID="5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c" exitCode=0 Nov 03 07:24:07.883304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:07.882874 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" event={"ID":"94f95955-229f-446c-ac9f-2fab451a3889","Type":"ContainerDied","Data":"5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c"} Nov 03 07:24:07.883304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:07.883240 4409 scope.go:122] "RemoveContainer" containerID="5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c" Nov 03 07:24:07.884755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:07.884698 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:24:07.884848 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:07.884793 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:24:07.884848 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:07.884828 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:24:08.800798 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:24:08.800755 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:24:08.896573 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:08.896538 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" event={"ID":"94f95955-229f-446c-ac9f-2fab451a3889","Type":"ContainerStarted","Data":"68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4"} Nov 03 07:24:09.090808 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:09.090669 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:24:09.091328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:09.091298 4409 patch_prober.go:29] interesting pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:24:09.091461 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:09.091368 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 07:24:10.089960 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:10.089929 4409 patch_prober.go:29] interesting pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:24:10.090506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:10.090437 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 07:24:11.084547 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:11.084511 4409 patch_prober.go:29] interesting pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:24:11.086998 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:11.086967 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 07:24:11.943042 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:11.943005 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5"] Nov 03 07:24:11.947233 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:11.947214 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.050010 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.049985 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5c169f6a-serving-cert\"" Nov 03 07:24:12.053953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.053929 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5"] Nov 03 07:24:12.087102 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.087069 4409 patch_prober.go:29] interesting pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:24:12.093851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.093815 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 07:24:12.114250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.114222 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-maas-api-tls\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.114574 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.114547 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c42291a-6e83-445f-86f7-1af9e7307338-openshift-service-ca\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.114756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.114734 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-metrics-certs\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.114851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.114835 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrlp\" (UniqueName: \"kubernetes.io/projected/8c42291a-6e83-445f-86f7-1af9e7307338-kube-api-access-zvrlp\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.219793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.219230 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c42291a-6e83-445f-86f7-1af9e7307338-openshift-service-ca\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.219793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.219386 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-metrics-certs\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.219793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.219460 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrlp\" (UniqueName: \"kubernetes.io/projected/8c42291a-6e83-445f-86f7-1af9e7307338-kube-api-access-zvrlp\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.219793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.219511 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-maas-api-tls\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.220132 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:24:12.219807 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-serving-cert: secret "maas-api-e2e-ait-5c169f6a-serving-cert" not found Nov 03 07:24:12.220132 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:24:12.219877 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-maas-api-tls podName:8c42291a-6e83-445f-86f7-1af9e7307338 nodeName:}" failed. No retries permitted until 2026-11-03 07:24:12.719853729 +0000 UTC m=+1507.912395462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-maas-api-tls") pod "maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" (UID: "8c42291a-6e83-445f-86f7-1af9e7307338") : secret "maas-api-e2e-ait-5c169f6a-serving-cert" not found Nov 03 07:24:12.223737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.220879 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c42291a-6e83-445f-86f7-1af9e7307338-openshift-service-ca\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.247891 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.243230 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-metrics-certs\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.329681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.329646 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrlp\" (UniqueName: \"kubernetes.io/projected/8c42291a-6e83-445f-86f7-1af9e7307338-kube-api-access-zvrlp\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.735519 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.735479 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-maas-api-tls\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.738295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.738267 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-maas-api-tls\") pod \"maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:12.861019 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:12.860987 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:13.083738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:13.079126 4409 patch_prober.go:29] interesting pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:24:13.083738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:13.079192 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 07:24:13.293850 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:13.293819 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5"] Nov 03 07:24:13.294039 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:24:13.294012 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c42291a_6e83_445f_86f7_1af9e7307338.slice/crio-0abef7362bc9b1f75e334e8b44376680e508c86f4f444fcdcb372e7e85a437ed WatchSource:0}: Error finding container 0abef7362bc9b1f75e334e8b44376680e508c86f4f444fcdcb372e7e85a437ed: Status 404 returned error can't find the container with id 0abef7362bc9b1f75e334e8b44376680e508c86f4f444fcdcb372e7e85a437ed Nov 03 07:24:13.538503 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:13.538422 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:24:13.934226 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:13.934189 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" event={"ID":"8c42291a-6e83-445f-86f7-1af9e7307338","Type":"ContainerStarted","Data":"0abef7362bc9b1f75e334e8b44376680e508c86f4f444fcdcb372e7e85a437ed"} Nov 03 07:24:14.077116 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:14.077070 4409 patch_prober.go:29] interesting pod/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 07:24:14.077285 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:14.077136 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 07:24:14.941987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:14.941945 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" event={"ID":"8c42291a-6e83-445f-86f7-1af9e7307338","Type":"ContainerStarted","Data":"b9c9dc5779bef400651a43d658fda0d98cd1696b1f4459e266f1a2e4f04ee668"} Nov 03 07:24:14.942452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:14.942005 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:15.070328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:15.070276 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" podStartSLOduration=3.283023551 podStartE2EDuration="4.070261908s" totalImagesPullingTime="787.238357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:24:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:24:15.067316648 +0000 UTC m=+1510.259858397" watchObservedRunningTime="2026-11-03 07:24:15.070261908 +0000 UTC m=+1510.262803658" Nov 03 07:24:15.081631 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:15.081609 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:24:15.082523 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:15.082508 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:24:18.861927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.861895 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6"] Nov 03 07:24:18.865925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.865900 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:18.920243 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.920222 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb"] Nov 03 07:24:18.923357 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.923340 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:18.944318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.944291 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:18.944417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.944363 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:18.944511 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.944498 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:18.944550 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.944527 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:18.944662 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.944650 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:18.944754 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.944686 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:18.944821 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.944763 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:18.944821 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.944798 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:18.944958 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.944932 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzqmh\" (UniqueName: \"kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-kube-api-access-jzqmh\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:18.968323 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:18.968301 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6"] Nov 03 07:24:19.026456 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.026418 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb"] Nov 03 07:24:19.046652 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.046628 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.046884 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.046661 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.046884 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.046682 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.046971 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.046908 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.047007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.046978 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.047324 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047063 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.047324 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047121 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.047324 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047237 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.047324 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047263 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.047324 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047296 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.047672 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047371 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgwnc\" (UniqueName: \"kubernetes.io/projected/2a99e5d6-cdd0-41f3-804d-5b478c92f432-kube-api-access-cgwnc\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.047672 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047412 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.047672 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047457 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.047672 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047487 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.047672 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047520 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.047672 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047563 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.047672 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047610 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqmh\" (UniqueName: \"kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-kube-api-access-jzqmh\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.047672 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047673 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.048159 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047753 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.048159 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047766 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.048159 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.047839 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.048324 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.048305 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.048392 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.048331 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.050313 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.050285 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.050399 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.050351 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.117842 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.117757 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzqmh\" (UniqueName: \"kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-kube-api-access-jzqmh\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.118532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.118478 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gp9f6\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.149442 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.149418 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.149442 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.149450 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.149654 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.149472 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.149654 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.149551 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.149654 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.149598 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.149893 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.149778 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.149893 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.149813 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.149893 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.149844 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.149893 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.149875 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgwnc\" (UniqueName: \"kubernetes.io/projected/2a99e5d6-cdd0-41f3-804d-5b478c92f432-kube-api-access-cgwnc\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.150116 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.149914 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.150116 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.150024 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.150563 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.150535 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.150691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.150604 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.150812 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.150696 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.152537 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.152499 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.152661 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.152638 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.178637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.178616 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:19.216078 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.216054 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a99e5d6-cdd0-41f3-804d-5b478c92f432-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.216275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.216252 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgwnc\" (UniqueName: \"kubernetes.io/projected/2a99e5d6-cdd0-41f3-804d-5b478c92f432-kube-api-access-cgwnc\") pod \"maas-default-gateway-openshift-default-58d898bd55-qrckb\" (UID: \"2a99e5d6-cdd0-41f3-804d-5b478c92f432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.237184 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.237155 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:19.612677 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.612648 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6"] Nov 03 07:24:19.648655 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:24:19.648629 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54db83b_2f0a_46d1_90de_84c048ff4e0d.slice/crio-fc728d6463d7e39f52897efba2499ce7fd97e403b8ffb6c3494b5ccd77ba3b68 WatchSource:0}: Error finding container fc728d6463d7e39f52897efba2499ce7fd97e403b8ffb6c3494b5ccd77ba3b68: Status 404 returned error can't find the container with id fc728d6463d7e39f52897efba2499ce7fd97e403b8ffb6c3494b5ccd77ba3b68 Nov 03 07:24:19.651286 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.651250 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:24:19.651371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.651316 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:24:19.651371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.651350 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:24:19.673787 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.673755 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb"] Nov 03 07:24:19.674941 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:24:19.674900 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a99e5d6_cdd0_41f3_804d_5b478c92f432.slice/crio-de2ab30fa7ae7f5efa3d2fdb14bcff3120aaa3bad5b42fb7aa28119d37a24664 WatchSource:0}: Error finding container de2ab30fa7ae7f5efa3d2fdb14bcff3120aaa3bad5b42fb7aa28119d37a24664: Status 404 returned error can't find the container with id de2ab30fa7ae7f5efa3d2fdb14bcff3120aaa3bad5b42fb7aa28119d37a24664 Nov 03 07:24:19.681724 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.681677 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:24:19.681816 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.681785 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:24:19.681860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.681826 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:24:19.982227 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.982107 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" event={"ID":"d54db83b-2f0a-46d1-90de-84c048ff4e0d","Type":"ContainerStarted","Data":"5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c"} Nov 03 07:24:19.982227 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.982181 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" event={"ID":"d54db83b-2f0a-46d1-90de-84c048ff4e0d","Type":"ContainerStarted","Data":"fc728d6463d7e39f52897efba2499ce7fd97e403b8ffb6c3494b5ccd77ba3b68"} Nov 03 07:24:19.984385 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.984354 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" event={"ID":"2a99e5d6-cdd0-41f3-804d-5b478c92f432","Type":"ContainerStarted","Data":"d5428e1537b761dc06b5e380315186b22ae2b0edd6b2bcaf31282546e64f72cc"} Nov 03 07:24:19.984524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:19.984411 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" event={"ID":"2a99e5d6-cdd0-41f3-804d-5b478c92f432","Type":"ContainerStarted","Data":"de2ab30fa7ae7f5efa3d2fdb14bcff3120aaa3bad5b42fb7aa28119d37a24664"} Nov 03 07:24:20.108406 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:20.108354 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" podStartSLOduration=2.108333958 podStartE2EDuration="2.108333958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:24:20.105615944 +0000 UTC m=+1515.298157693" watchObservedRunningTime="2026-11-03 07:24:20.108333958 +0000 UTC m=+1515.300875770" Nov 03 07:24:20.179060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:20.179024 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:20.180668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:20.180638 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 07:24:20.180802 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:20.180699 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 07:24:20.230777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:20.230645 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" podStartSLOduration=2.230626034 podStartE2EDuration="2.230626034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:24:20.227890904 +0000 UTC m=+1515.420432679" watchObservedRunningTime="2026-11-03 07:24:20.230626034 +0000 UTC m=+1515.423167785" Nov 03 07:24:20.238539 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:20.238444 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:20.240253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:20.240223 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qrckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 07:24:20.240419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:20.240293 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" podUID="2a99e5d6-cdd0-41f3-804d-5b478c92f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 07:24:20.954373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:20.954347 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:24:21.195739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:21.194213 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 07:24:21.195739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:21.194274 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 07:24:21.247040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:21.245972 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qrckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 07:24:21.247040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:21.246028 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" podUID="2a99e5d6-cdd0-41f3-804d-5b478c92f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 07:24:22.184738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:22.182790 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 07:24:22.184738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:22.182849 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 07:24:22.238838 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:22.238802 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qrckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 07:24:22.239221 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:22.238856 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" podUID="2a99e5d6-cdd0-41f3-804d-5b478c92f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 07:24:23.184741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:23.180804 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 07:24:23.184741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:23.180867 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 07:24:23.238354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:23.238320 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qrckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 07:24:23.238498 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:23.238370 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" podUID="2a99e5d6-cdd0-41f3-804d-5b478c92f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 07:24:24.181100 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:24.181060 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 07:24:24.181702 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:24.181666 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 07:24:24.238512 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:24.238484 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qrckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 07:24:24.238780 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:24.238751 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" podUID="2a99e5d6-cdd0-41f3-804d-5b478c92f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 07:24:25.186741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:25.185077 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 07:24:25.186741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:25.185138 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 07:24:25.238477 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:25.238445 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qrckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 07:24:25.238644 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:25.238497 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" podUID="2a99e5d6-cdd0-41f3-804d-5b478c92f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 07:24:26.179897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:26.179861 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 07:24:26.180102 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:26.179922 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 07:24:26.238413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:26.238382 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qrckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 07:24:26.238829 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:26.238444 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" podUID="2a99e5d6-cdd0-41f3-804d-5b478c92f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 07:24:27.188420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:27.188381 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 07:24:27.188623 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:27.188433 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 07:24:27.237724 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:27.237692 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qrckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 07:24:27.237870 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:27.237771 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" podUID="2a99e5d6-cdd0-41f3-804d-5b478c92f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 07:24:28.380510 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:28.380475 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:28.382130 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:28.382104 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:28.383432 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:28.383408 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:24:28.426496 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:28.425887 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:28.427401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:28.426704 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:28.427980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:28.427960 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" Nov 03 07:24:38.800641 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:24:38.800594 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:24:45.082880 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:45.082829 4409 patch_prober.go:29] interesting pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:24:45.083440 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:45.082916 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:24:46.350954 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:46.350921 4409 generic.go:356] "Generic (PLEG): container finished" podID="80f64254-ffd6-4900-94fa-c6386145579a" containerID="8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b" exitCode=0 Nov 03 07:24:46.351392 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:46.350993 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" event={"ID":"80f64254-ffd6-4900-94fa-c6386145579a","Type":"ContainerDied","Data":"8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b"} Nov 03 07:24:46.351476 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:46.351455 4409 scope.go:122] "RemoveContainer" containerID="8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b" Nov 03 07:24:46.353046 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:46.353015 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:24:46.353169 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:46.353101 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:24:46.353169 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:46.353154 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:24:47.361074 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:47.361037 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" event={"ID":"80f64254-ffd6-4900-94fa-c6386145579a","Type":"ContainerStarted","Data":"af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124"} Nov 03 07:24:48.074738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:48.072108 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:24:48.074738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:48.072246 4409 patch_prober.go:29] interesting pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:24:48.074738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:48.072291 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 07:24:49.071740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:49.064792 4409 patch_prober.go:29] interesting pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:24:49.071740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:49.064864 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 07:24:50.070286 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:50.070247 4409 patch_prober.go:29] interesting pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:24:50.070569 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:50.070540 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 07:24:51.087738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:51.076035 4409 patch_prober.go:29] interesting pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:24:51.087738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:51.076096 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 07:24:52.085742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:52.083830 4409 patch_prober.go:29] interesting pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 07:24:52.085742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:52.083900 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 07:24:53.069253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:53.069224 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:24:53.069706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:53.069691 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:24:53.070422 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:53.070405 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:24:55.308806 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:24:55.308665 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2"] Nov 03 07:25:00.143441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:00.143410 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml"] Nov 03 07:25:00.252745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:00.252689 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf"] Nov 03 07:25:00.320145 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:00.320112 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" podUID="c38eead0-09aa-49ef-af09-1fafd54aec6b" containerName="payload-pre-processing" containerID="cri-o://e40a44001bea0e34d2c9903f28f70dc38270f6081f100ecdd859f30b03b9bd8a" gracePeriod=25 Nov 03 07:25:00.521740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:00.520633 4409 generic.go:356] "Generic (PLEG): container finished" podID="c38eead0-09aa-49ef-af09-1fafd54aec6b" containerID="e40a44001bea0e34d2c9903f28f70dc38270f6081f100ecdd859f30b03b9bd8a" exitCode=0 Nov 03 07:25:00.521740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:00.520677 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" event={"ID":"c38eead0-09aa-49ef-af09-1fafd54aec6b","Type":"ContainerDied","Data":"e40a44001bea0e34d2c9903f28f70dc38270f6081f100ecdd859f30b03b9bd8a"} Nov 03 07:25:00.898408 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:00.898384 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" Nov 03 07:25:00.991301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:00.991270 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c38eead0-09aa-49ef-af09-1fafd54aec6b-kube-api-access-d86cd\" (UniqueName: \"kubernetes.io/projected/c38eead0-09aa-49ef-af09-1fafd54aec6b-kube-api-access-d86cd\") pod \"c38eead0-09aa-49ef-af09-1fafd54aec6b\" (UID: \"c38eead0-09aa-49ef-af09-1fafd54aec6b\") " Nov 03 07:25:00.991462 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:00.991345 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38eead0-09aa-49ef-af09-1fafd54aec6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38eead0-09aa-49ef-af09-1fafd54aec6b-plugins-config-volume\") pod \"c38eead0-09aa-49ef-af09-1fafd54aec6b\" (UID: \"c38eead0-09aa-49ef-af09-1fafd54aec6b\") " Nov 03 07:25:00.991740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:00.991676 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38eead0-09aa-49ef-af09-1fafd54aec6b-plugins-config-volume" pod "c38eead0-09aa-49ef-af09-1fafd54aec6b" (UID: "c38eead0-09aa-49ef-af09-1fafd54aec6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:25:00.992088 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:00.992065 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38eead0-09aa-49ef-af09-1fafd54aec6b-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:00.994396 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:00.994375 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38eead0-09aa-49ef-af09-1fafd54aec6b-kube-api-access-d86cd" pod "c38eead0-09aa-49ef-af09-1fafd54aec6b" (UID: "c38eead0-09aa-49ef-af09-1fafd54aec6b"). InnerVolumeSpecName "kube-api-access-d86cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:25:01.094260 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:01.094227 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d86cd\" (UniqueName: \"kubernetes.io/projected/c38eead0-09aa-49ef-af09-1fafd54aec6b-kube-api-access-d86cd\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:01.529514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:01.529474 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" event={"ID":"c38eead0-09aa-49ef-af09-1fafd54aec6b","Type":"ContainerDied","Data":"f075b685f067031de40eb11bab112570a2128e0725c84ed662f81b3219226683"} Nov 03 07:25:01.529972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:01.529486 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2" Nov 03 07:25:01.529972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:01.529575 4409 scope.go:122] "RemoveContainer" containerID="e40a44001bea0e34d2c9903f28f70dc38270f6081f100ecdd859f30b03b9bd8a" Nov 03 07:25:01.772144 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:01.772105 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2"] Nov 03 07:25:01.833799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:01.817612 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67db4bc0-74d658899f-224v2"] Nov 03 07:25:02.915430 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:02.915396 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c38eead0-09aa-49ef-af09-1fafd54aec6b" path="/var/lib/kubelet/pods/c38eead0-09aa-49ef-af09-1fafd54aec6b/volumes" Nov 03 07:25:05.144660 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.144622 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" podUID="e71b9dc9-00d9-43bd-8020-503fe826d1cc" containerName="payload-processing" containerID="cri-o://06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541" gracePeriod=25 Nov 03 07:25:05.253115 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.253061 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" podUID="bf62d1fb-e609-40b2-b232-5c2ed3eefcdc" containerName="payload-pre-processing" containerID="cri-o://1cdaceb91a5e4c237ea738dc4bbb3209685cd20124572d0f10249f110d93ea41" gracePeriod=25 Nov 03 07:25:05.546900 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.540910 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" Nov 03 07:25:05.605862 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.605763 4409 generic.go:356] "Generic (PLEG): container finished" podID="e71b9dc9-00d9-43bd-8020-503fe826d1cc" containerID="06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541" exitCode=0 Nov 03 07:25:05.606014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.605876 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" Nov 03 07:25:05.606014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.605889 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" event={"ID":"e71b9dc9-00d9-43bd-8020-503fe826d1cc","Type":"ContainerDied","Data":"06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541"} Nov 03 07:25:05.606014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.605940 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml" event={"ID":"e71b9dc9-00d9-43bd-8020-503fe826d1cc","Type":"ContainerDied","Data":"6e951e89c5da8e8871ac0914467fb4c800799c29eef09a3bb795877e764971de"} Nov 03 07:25:05.606014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.605969 4409 scope.go:122] "RemoveContainer" containerID="06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541" Nov 03 07:25:05.608246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.608219 4409 generic.go:356] "Generic (PLEG): container finished" podID="bf62d1fb-e609-40b2-b232-5c2ed3eefcdc" containerID="1cdaceb91a5e4c237ea738dc4bbb3209685cd20124572d0f10249f110d93ea41" exitCode=0 Nov 03 07:25:05.608409 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.608356 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" event={"ID":"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc","Type":"ContainerDied","Data":"1cdaceb91a5e4c237ea738dc4bbb3209685cd20124572d0f10249f110d93ea41"} Nov 03 07:25:05.619361 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.619338 4409 scope.go:122] "RemoveContainer" containerID="06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541" Nov 03 07:25:05.619682 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:25:05.619658 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541\": container with ID starting with 06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541 not found: ID does not exist" containerID="06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541" Nov 03 07:25:05.619827 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.619694 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541"} err="failed to get container status \"06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541\": rpc error: code = NotFound desc = could not find container \"06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541\": container with ID starting with 06aa400cf6c3465166da062cc88a9b31a275b179d45c60aa253d5d7a4592b541 not found: ID does not exist" Nov 03 07:25:05.675682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.675663 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" Nov 03 07:25:05.685567 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.685539 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71b9dc9-00d9-43bd-8020-503fe826d1cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71b9dc9-00d9-43bd-8020-503fe826d1cc-plugins-config-volume\") pod \"e71b9dc9-00d9-43bd-8020-503fe826d1cc\" (UID: \"e71b9dc9-00d9-43bd-8020-503fe826d1cc\") " Nov 03 07:25:05.685777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.685753 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71b9dc9-00d9-43bd-8020-503fe826d1cc-kube-api-access-hrs8k\" (UniqueName: \"kubernetes.io/projected/e71b9dc9-00d9-43bd-8020-503fe826d1cc-kube-api-access-hrs8k\") pod \"e71b9dc9-00d9-43bd-8020-503fe826d1cc\" (UID: \"e71b9dc9-00d9-43bd-8020-503fe826d1cc\") " Nov 03 07:25:05.686307 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.686281 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71b9dc9-00d9-43bd-8020-503fe826d1cc-plugins-config-volume" pod "e71b9dc9-00d9-43bd-8020-503fe826d1cc" (UID: "e71b9dc9-00d9-43bd-8020-503fe826d1cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:25:05.686524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.686507 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71b9dc9-00d9-43bd-8020-503fe826d1cc-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:05.688880 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.688856 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71b9dc9-00d9-43bd-8020-503fe826d1cc-kube-api-access-hrs8k" pod "e71b9dc9-00d9-43bd-8020-503fe826d1cc" (UID: "e71b9dc9-00d9-43bd-8020-503fe826d1cc"). InnerVolumeSpecName "kube-api-access-hrs8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:25:05.787196 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.787168 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-plugins-config-volume\") pod \"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc\" (UID: \"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc\") " Nov 03 07:25:05.787338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.787252 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-kube-api-access-86xq8\" (UniqueName: \"kubernetes.io/projected/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-kube-api-access-86xq8\") pod \"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc\" (UID: \"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc\") " Nov 03 07:25:05.787756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.787686 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-plugins-config-volume" pod "bf62d1fb-e609-40b2-b232-5c2ed3eefcdc" (UID: "bf62d1fb-e609-40b2-b232-5c2ed3eefcdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:25:05.788079 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.787961 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrs8k\" (UniqueName: \"kubernetes.io/projected/e71b9dc9-00d9-43bd-8020-503fe826d1cc-kube-api-access-hrs8k\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:05.788079 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.787984 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:05.789936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.789909 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-kube-api-access-86xq8" pod "bf62d1fb-e609-40b2-b232-5c2ed3eefcdc" (UID: "bf62d1fb-e609-40b2-b232-5c2ed3eefcdc"). InnerVolumeSpecName "kube-api-access-86xq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:25:05.889885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:05.889853 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86xq8\" (UniqueName: \"kubernetes.io/projected/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc-kube-api-access-86xq8\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:06.034883 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:06.034857 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml"] Nov 03 07:25:06.039434 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:06.039410 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-33a46b37-5d8c64fb59-bvpml"] Nov 03 07:25:06.617165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:06.617138 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" event={"ID":"bf62d1fb-e609-40b2-b232-5c2ed3eefcdc","Type":"ContainerDied","Data":"5d1b09509336aeb70a85a5cfe92d13dc6f0d5dc8c2b4ce25c3cdd603e038c148"} Nov 03 07:25:06.617538 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:06.617190 4409 scope.go:122] "RemoveContainer" containerID="1cdaceb91a5e4c237ea738dc4bbb3209685cd20124572d0f10249f110d93ea41" Nov 03 07:25:06.617538 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:06.617152 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf" Nov 03 07:25:06.789414 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:06.789387 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf"] Nov 03 07:25:06.838823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:06.838757 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-33a46b37-5cb99877bc-7cpqf"] Nov 03 07:25:06.914588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:06.914556 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf62d1fb-e609-40b2-b232-5c2ed3eefcdc" path="/var/lib/kubelet/pods/bf62d1fb-e609-40b2-b232-5c2ed3eefcdc/volumes" Nov 03 07:25:06.915161 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:06.915144 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71b9dc9-00d9-43bd-8020-503fe826d1cc" path="/var/lib/kubelet/pods/e71b9dc9-00d9-43bd-8020-503fe826d1cc/volumes" Nov 03 07:25:08.774153 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:25:08.774117 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:25:30.700593 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:30.700558 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt"] Nov 03 07:25:30.701169 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:30.700761 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" podUID="29b881ff-f8e3-4af0-9f26-2fbe5720825b" containerName="maas-api" containerID="cri-o://13251e72d3091799b018efd7c69081797debd5736af0070058e78047834af9b6" gracePeriod=30 Nov 03 07:25:30.839671 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:30.839638 4409 generic.go:356] "Generic (PLEG): container finished" podID="29b881ff-f8e3-4af0-9f26-2fbe5720825b" containerID="13251e72d3091799b018efd7c69081797debd5736af0070058e78047834af9b6" exitCode=0 Nov 03 07:25:30.839865 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:30.839734 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" event={"ID":"29b881ff-f8e3-4af0-9f26-2fbe5720825b","Type":"ContainerDied","Data":"13251e72d3091799b018efd7c69081797debd5736af0070058e78047834af9b6"} Nov 03 07:25:31.338961 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.338922 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:25:31.386687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.385561 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8"] Nov 03 07:25:31.391884 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.391853 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t"] Nov 03 07:25:31.401826 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.401790 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-metrics-certs\") pod \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " Nov 03 07:25:31.401953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.401854 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-maas-api-tls\") pod \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " Nov 03 07:25:31.401998 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.401961 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b881ff-f8e3-4af0-9f26-2fbe5720825b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b881ff-f8e3-4af0-9f26-2fbe5720825b-openshift-service-ca\") pod \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " Nov 03 07:25:31.402115 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.402071 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b881ff-f8e3-4af0-9f26-2fbe5720825b-kube-api-access-cvzfj\" (UniqueName: \"kubernetes.io/projected/29b881ff-f8e3-4af0-9f26-2fbe5720825b-kube-api-access-cvzfj\") pod \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\" (UID: \"29b881ff-f8e3-4af0-9f26-2fbe5720825b\") " Nov 03 07:25:31.402423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.402396 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b881ff-f8e3-4af0-9f26-2fbe5720825b-openshift-service-ca" pod "29b881ff-f8e3-4af0-9f26-2fbe5720825b" (UID: "29b881ff-f8e3-4af0-9f26-2fbe5720825b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:25:31.402875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.402804 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b881ff-f8e3-4af0-9f26-2fbe5720825b-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:31.404143 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.404122 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-maas-api-tls" pod "29b881ff-f8e3-4af0-9f26-2fbe5720825b" (UID: "29b881ff-f8e3-4af0-9f26-2fbe5720825b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:25:31.404238 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.404156 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-metrics-certs" pod "29b881ff-f8e3-4af0-9f26-2fbe5720825b" (UID: "29b881ff-f8e3-4af0-9f26-2fbe5720825b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:25:31.404319 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.404305 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b881ff-f8e3-4af0-9f26-2fbe5720825b-kube-api-access-cvzfj" pod "29b881ff-f8e3-4af0-9f26-2fbe5720825b" (UID: "29b881ff-f8e3-4af0-9f26-2fbe5720825b"). InnerVolumeSpecName "kube-api-access-cvzfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:25:31.504605 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.504534 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:31.504605 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.504561 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b881ff-f8e3-4af0-9f26-2fbe5720825b-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:31.504605 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:31.504571 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvzfj\" (UniqueName: \"kubernetes.io/projected/29b881ff-f8e3-4af0-9f26-2fbe5720825b-kube-api-access-cvzfj\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:32.078460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.060218 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv"] Nov 03 07:25:32.078460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.060634 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" podUID="d40c18a0-45b8-4612-a17d-b18c0599623b" containerName="maas-api" containerID="cri-o://cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed" gracePeriod=30 Nov 03 07:25:32.123803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.123769 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" event={"ID":"29b881ff-f8e3-4af0-9f26-2fbe5720825b","Type":"ContainerDied","Data":"28cc65389cc3a966f3b72bb3e63b2d234de434834d77c5f732ffdb68b36037ad"} Nov 03 07:25:32.124134 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.124113 4409 scope.go:122] "RemoveContainer" containerID="13251e72d3091799b018efd7c69081797debd5736af0070058e78047834af9b6" Nov 03 07:25:32.124581 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.124563 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" Nov 03 07:25:32.447796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.440758 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt"] Nov 03 07:25:32.459739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.455685 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt"] Nov 03 07:25:32.685005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.684984 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:25:32.742375 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.742318 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-metrics-certs\") pod \"d40c18a0-45b8-4612-a17d-b18c0599623b\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " Nov 03 07:25:32.742508 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.742406 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40c18a0-45b8-4612-a17d-b18c0599623b-kube-api-access-8ljhw\" (UniqueName: \"kubernetes.io/projected/d40c18a0-45b8-4612-a17d-b18c0599623b-kube-api-access-8ljhw\") pod \"d40c18a0-45b8-4612-a17d-b18c0599623b\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " Nov 03 07:25:32.742556 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.742515 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40c18a0-45b8-4612-a17d-b18c0599623b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d40c18a0-45b8-4612-a17d-b18c0599623b-openshift-service-ca\") pod \"d40c18a0-45b8-4612-a17d-b18c0599623b\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " Nov 03 07:25:32.742754 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.742733 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-maas-api-tls\") pod \"d40c18a0-45b8-4612-a17d-b18c0599623b\" (UID: \"d40c18a0-45b8-4612-a17d-b18c0599623b\") " Nov 03 07:25:32.742943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.742921 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40c18a0-45b8-4612-a17d-b18c0599623b-openshift-service-ca" pod "d40c18a0-45b8-4612-a17d-b18c0599623b" (UID: "d40c18a0-45b8-4612-a17d-b18c0599623b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:25:32.743360 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.743337 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d40c18a0-45b8-4612-a17d-b18c0599623b-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:32.744853 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.744829 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-maas-api-tls" pod "d40c18a0-45b8-4612-a17d-b18c0599623b" (UID: "d40c18a0-45b8-4612-a17d-b18c0599623b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:25:32.744853 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.744846 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-metrics-certs" pod "d40c18a0-45b8-4612-a17d-b18c0599623b" (UID: "d40c18a0-45b8-4612-a17d-b18c0599623b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:25:32.745004 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.744861 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40c18a0-45b8-4612-a17d-b18c0599623b-kube-api-access-8ljhw" pod "d40c18a0-45b8-4612-a17d-b18c0599623b" (UID: "d40c18a0-45b8-4612-a17d-b18c0599623b"). InnerVolumeSpecName "kube-api-access-8ljhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:25:32.852744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.852444 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:32.852744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.852481 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d40c18a0-45b8-4612-a17d-b18c0599623b-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:32.852744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:32.852501 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ljhw\" (UniqueName: \"kubernetes.io/projected/d40c18a0-45b8-4612-a17d-b18c0599623b-kube-api-access-8ljhw\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:33.173743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:33.149614 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b881ff-f8e3-4af0-9f26-2fbe5720825b" path="/var/lib/kubelet/pods/29b881ff-f8e3-4af0-9f26-2fbe5720825b/volumes" Nov 03 07:25:33.266735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:33.258234 4409 generic.go:356] "Generic (PLEG): container finished" podID="d40c18a0-45b8-4612-a17d-b18c0599623b" containerID="cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed" exitCode=0 Nov 03 07:25:33.266735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:33.258495 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" Nov 03 07:25:33.319085 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:33.319058 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd"] Nov 03 07:25:33.319331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:33.319308 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" event={"ID":"d40c18a0-45b8-4612-a17d-b18c0599623b","Type":"ContainerDied","Data":"cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed"} Nov 03 07:25:33.319449 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:33.319434 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv" event={"ID":"d40c18a0-45b8-4612-a17d-b18c0599623b","Type":"ContainerDied","Data":"efddfb282f18b46ebb601c04acdab6372b0cd640042ab90a63113269c40bfa1b"} Nov 03 07:25:33.319582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:33.319569 4409 scope.go:122] "RemoveContainer" containerID="cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed" Nov 03 07:25:33.367105 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:33.323901 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" containerID="cri-o://68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4" gracePeriod=30 Nov 03 07:25:33.691740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:33.689925 4409 scope.go:122] "RemoveContainer" containerID="cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed" Nov 03 07:25:33.703956 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:25:33.700377 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed\": container with ID starting with cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed not found: ID does not exist" containerID="cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed" Nov 03 07:25:33.703956 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:33.700425 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed"} err="failed to get container status \"cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed\": rpc error: code = NotFound desc = could not find container \"cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed\": container with ID starting with cc501e95da8242270012a3634597c76d5964d7ec7cd64550089e5dda3f2004ed not found: ID does not exist" Nov 03 07:25:33.710751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:33.710658 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv"] Nov 03 07:25:33.718217 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:33.718171 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-33a46b37-7f65cc7d5f-tmmnv"] Nov 03 07:25:34.281921 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.281877 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67db4bc0-77766bbb5c-qx9pt" podUID="29b881ff-f8e3-4af0-9f26-2fbe5720825b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.88:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:25:34.461032 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461002 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr"] Nov 03 07:25:34.461673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461660 4409 cpu_manager.go:414] "containerMap: removing container" podUID="d40c18a0-45b8-4612-a17d-b18c0599623b" containerName="maas-api" Nov 03 07:25:34.461764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461675 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40c18a0-45b8-4612-a17d-b18c0599623b" containerName="maas-api" Nov 03 07:25:34.461764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461692 4409 cpu_manager.go:414] "containerMap: removing container" podUID="c38eead0-09aa-49ef-af09-1fafd54aec6b" containerName="payload-pre-processing" Nov 03 07:25:34.461764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461697 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38eead0-09aa-49ef-af09-1fafd54aec6b" containerName="payload-pre-processing" Nov 03 07:25:34.461764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461746 4409 cpu_manager.go:414] "containerMap: removing container" podUID="bf62d1fb-e609-40b2-b232-5c2ed3eefcdc" containerName="payload-pre-processing" Nov 03 07:25:34.461764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461752 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf62d1fb-e609-40b2-b232-5c2ed3eefcdc" containerName="payload-pre-processing" Nov 03 07:25:34.461764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461763 4409 cpu_manager.go:414] "containerMap: removing container" podUID="29b881ff-f8e3-4af0-9f26-2fbe5720825b" containerName="maas-api" Nov 03 07:25:34.461962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461769 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b881ff-f8e3-4af0-9f26-2fbe5720825b" containerName="maas-api" Nov 03 07:25:34.461962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461779 4409 cpu_manager.go:414] "containerMap: removing container" podUID="e71b9dc9-00d9-43bd-8020-503fe826d1cc" containerName="payload-processing" Nov 03 07:25:34.461962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461785 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71b9dc9-00d9-43bd-8020-503fe826d1cc" containerName="payload-processing" Nov 03 07:25:34.461962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461891 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf62d1fb-e609-40b2-b232-5c2ed3eefcdc" containerName="payload-pre-processing" Nov 03 07:25:34.461962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461902 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b881ff-f8e3-4af0-9f26-2fbe5720825b" containerName="maas-api" Nov 03 07:25:34.461962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461910 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71b9dc9-00d9-43bd-8020-503fe826d1cc" containerName="payload-processing" Nov 03 07:25:34.461962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461915 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40c18a0-45b8-4612-a17d-b18c0599623b" containerName="maas-api" Nov 03 07:25:34.461962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.461922 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="c38eead0-09aa-49ef-af09-1fafd54aec6b" containerName="payload-pre-processing" Nov 03 07:25:34.467607 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.466215 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.535783 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.527082 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6d089af5-openshift-default-dockercfg-ghlh4\"" Nov 03 07:25:34.698966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.678640 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-podinfo\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.698966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.678690 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d107dd0-ba31-4e3c-80fb-377d91700524-istiod-ca-cert\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.698966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.678771 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzm98\" (UniqueName: \"kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-kube-api-access-tzm98\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.698966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.678810 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-certs\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.698966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.678924 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-data\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.698966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.679035 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-socket\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.698966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.679090 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-credential-socket\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.698966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.679235 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-envoy\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.698966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.679274 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-token\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.787303 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.781736 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-data\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.787303 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.781878 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-socket\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.787303 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.781941 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-credential-socket\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.787303 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.782081 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-envoy\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.787303 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.782117 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-token\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.787303 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.782162 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-podinfo\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.787303 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.782185 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d107dd0-ba31-4e3c-80fb-377d91700524-istiod-ca-cert\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.787303 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.782240 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzm98\" (UniqueName: \"kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-kube-api-access-tzm98\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.787303 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.782276 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-certs\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.818735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.800922 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-certs\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.818735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.801633 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d107dd0-ba31-4e3c-80fb-377d91700524-istiod-ca-cert\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.818735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.816274 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-credential-socket\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.818735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.816928 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-data\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.822736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.821227 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-socket\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.841971 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.839375 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-podinfo\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.863741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.849786 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr"] Nov 03 07:25:34.863741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.850031 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-envoy\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:34.967739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.967452 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40c18a0-45b8-4612-a17d-b18c0599623b" path="/var/lib/kubelet/pods/d40c18a0-45b8-4612-a17d-b18c0599623b/volumes" Nov 03 07:25:35.000805 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:34.997935 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzm98\" (UniqueName: \"kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-kube-api-access-tzm98\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:35.005221 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:35.005188 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-token\") pod \"e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:35.091736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:35.086664 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:35.679065 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:35.677570 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr"] Nov 03 07:25:35.713079 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:25:35.713018 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d107dd0_ba31_4e3c_80fb_377d91700524.slice/crio-8b553cb9d3cee29fff836eb6b9159f664a9b50565c7901ba73596ee99ccc08e2 WatchSource:0}: Error finding container 8b553cb9d3cee29fff836eb6b9159f664a9b50565c7901ba73596ee99ccc08e2: Status 404 returned error can't find the container with id 8b553cb9d3cee29fff836eb6b9159f664a9b50565c7901ba73596ee99ccc08e2 Nov 03 07:25:35.732279 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:35.718284 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:25:35.732279 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:35.718378 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:25:35.732279 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:35.718420 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:25:36.287127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.287005 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" event={"ID":"9d107dd0-ba31-4e3c-80fb-377d91700524","Type":"ContainerStarted","Data":"77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f"} Nov 03 07:25:36.287127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.287068 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" event={"ID":"9d107dd0-ba31-4e3c-80fb-377d91700524","Type":"ContainerStarted","Data":"8b553cb9d3cee29fff836eb6b9159f664a9b50565c7901ba73596ee99ccc08e2"} Nov 03 07:25:36.325844 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.325806 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl"] Nov 03 07:25:36.329076 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.329057 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.384494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.384452 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-209c4c70-openshift-default-dockercfg-mkn2h\"" Nov 03 07:25:36.386580 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.386549 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" podUID="027e5dc0-f742-4388-a8dd-62de2f7651d2" containerName="payload-processing" containerID="cri-o://bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e" gracePeriod=25 Nov 03 07:25:36.393426 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.393258 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" podUID="2d42febe-a8a4-4b3c-b449-e19b9c957409" containerName="payload-pre-processing" containerID="cri-o://58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6" gracePeriod=25 Nov 03 07:25:36.424578 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.424548 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl"] Nov 03 07:25:36.443895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.443854 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" podStartSLOduration=2.443839366 podStartE2EDuration="2.443839366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:25:36.441331522 +0000 UTC m=+1591.633873291" watchObservedRunningTime="2026-11-03 07:25:36.443839366 +0000 UTC m=+1591.636381116" Nov 03 07:25:36.472010 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.471976 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-data\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.472010 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.472012 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-podinfo\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.472208 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.472181 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istiod-ca-cert\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.472244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.472220 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-credential-socket\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.472346 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.472325 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-token\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.472386 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.472373 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-socket\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.472447 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.472430 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-envoy\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.472518 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.472506 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9grbv\" (UniqueName: \"kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-kube-api-access-9grbv\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.472609 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.472598 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-certs\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.594740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.591843 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-envoy\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.594740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.591920 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9grbv\" (UniqueName: \"kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-kube-api-access-9grbv\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.594740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.592034 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-certs\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.594740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.592154 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-data\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.594740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.592179 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-podinfo\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.594740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.592318 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istiod-ca-cert\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.594740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.592355 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-credential-socket\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.594740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.592422 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-token\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.594740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.592465 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-socket\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.594740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.593009 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-socket\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.623743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.619247 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-certs\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.650925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.624242 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-credential-socket\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.650925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.624952 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istiod-ca-cert\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.650925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.650857 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-data\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.673756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.663410 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-podinfo\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.800735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.797735 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9grbv\" (UniqueName: \"kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-kube-api-access-9grbv\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.800735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.798451 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-envoy\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.813749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.802834 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-token\") pod \"e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:36.956330 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:36.945696 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:37.012601 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.012570 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" Nov 03 07:25:37.052769 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.052539 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" Nov 03 07:25:37.087827 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.087791 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:37.092981 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.092959 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:37.137357 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.137289 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027e5dc0-f742-4388-a8dd-62de2f7651d2-kube-api-access-w9bkr\" (UniqueName: \"kubernetes.io/projected/027e5dc0-f742-4388-a8dd-62de2f7651d2-kube-api-access-w9bkr\") pod \"027e5dc0-f742-4388-a8dd-62de2f7651d2\" (UID: \"027e5dc0-f742-4388-a8dd-62de2f7651d2\") " Nov 03 07:25:37.137502 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.137482 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d42febe-a8a4-4b3c-b449-e19b9c957409-kube-api-access-fqf2h\" (UniqueName: \"kubernetes.io/projected/2d42febe-a8a4-4b3c-b449-e19b9c957409-kube-api-access-fqf2h\") pod \"2d42febe-a8a4-4b3c-b449-e19b9c957409\" (UID: \"2d42febe-a8a4-4b3c-b449-e19b9c957409\") " Nov 03 07:25:37.137669 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.137648 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027e5dc0-f742-4388-a8dd-62de2f7651d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027e5dc0-f742-4388-a8dd-62de2f7651d2-plugins-config-volume\") pod \"027e5dc0-f742-4388-a8dd-62de2f7651d2\" (UID: \"027e5dc0-f742-4388-a8dd-62de2f7651d2\") " Nov 03 07:25:37.137878 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.137707 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d42febe-a8a4-4b3c-b449-e19b9c957409-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d42febe-a8a4-4b3c-b449-e19b9c957409-plugins-config-volume\") pod \"2d42febe-a8a4-4b3c-b449-e19b9c957409\" (UID: \"2d42febe-a8a4-4b3c-b449-e19b9c957409\") " Nov 03 07:25:37.138590 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.138159 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d42febe-a8a4-4b3c-b449-e19b9c957409-plugins-config-volume" pod "2d42febe-a8a4-4b3c-b449-e19b9c957409" (UID: "2d42febe-a8a4-4b3c-b449-e19b9c957409"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:25:37.138590 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.138543 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d42febe-a8a4-4b3c-b449-e19b9c957409-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:37.139168 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.139132 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027e5dc0-f742-4388-a8dd-62de2f7651d2-plugins-config-volume" pod "027e5dc0-f742-4388-a8dd-62de2f7651d2" (UID: "027e5dc0-f742-4388-a8dd-62de2f7651d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:25:37.140333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.140306 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d42febe-a8a4-4b3c-b449-e19b9c957409-kube-api-access-fqf2h" pod "2d42febe-a8a4-4b3c-b449-e19b9c957409" (UID: "2d42febe-a8a4-4b3c-b449-e19b9c957409"). InnerVolumeSpecName "kube-api-access-fqf2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:25:37.141393 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.141366 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027e5dc0-f742-4388-a8dd-62de2f7651d2-kube-api-access-w9bkr" pod "027e5dc0-f742-4388-a8dd-62de2f7651d2" (UID: "027e5dc0-f742-4388-a8dd-62de2f7651d2"). InnerVolumeSpecName "kube-api-access-w9bkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:25:37.240492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.240456 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027e5dc0-f742-4388-a8dd-62de2f7651d2-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:37.240492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.240484 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9bkr\" (UniqueName: \"kubernetes.io/projected/027e5dc0-f742-4388-a8dd-62de2f7651d2-kube-api-access-w9bkr\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:37.240492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.240496 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqf2h\" (UniqueName: \"kubernetes.io/projected/2d42febe-a8a4-4b3c-b449-e19b9c957409-kube-api-access-fqf2h\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:37.307725 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.307670 4409 generic.go:356] "Generic (PLEG): container finished" podID="027e5dc0-f742-4388-a8dd-62de2f7651d2" containerID="bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e" exitCode=0 Nov 03 07:25:37.308284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.307967 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" event={"ID":"027e5dc0-f742-4388-a8dd-62de2f7651d2","Type":"ContainerDied","Data":"bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e"} Nov 03 07:25:37.308284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.308043 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" Nov 03 07:25:37.308284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.308074 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8" event={"ID":"027e5dc0-f742-4388-a8dd-62de2f7651d2","Type":"ContainerDied","Data":"a19f9cd6dc190351d85a514fe54b77b47c7500b4b17dd4acb6d7280c254a0968"} Nov 03 07:25:37.308284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.308113 4409 scope.go:122] "RemoveContainer" containerID="bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e" Nov 03 07:25:37.310619 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.310597 4409 generic.go:356] "Generic (PLEG): container finished" podID="2d42febe-a8a4-4b3c-b449-e19b9c957409" containerID="58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6" exitCode=0 Nov 03 07:25:37.310783 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.310765 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" Nov 03 07:25:37.310851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.310809 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" event={"ID":"2d42febe-a8a4-4b3c-b449-e19b9c957409","Type":"ContainerDied","Data":"58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6"} Nov 03 07:25:37.310908 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.310855 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t" event={"ID":"2d42febe-a8a4-4b3c-b449-e19b9c957409","Type":"ContainerDied","Data":"9b2e8ae669b5f8caf668adc604579c34b93404589aed1888083d896a0bb53914"} Nov 03 07:25:37.311296 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.311277 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:37.313003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.312983 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:25:37.322179 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.322163 4409 scope.go:122] "RemoveContainer" containerID="bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e" Nov 03 07:25:37.322463 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:25:37.322444 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e\": container with ID starting with bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e not found: ID does not exist" containerID="bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e" Nov 03 07:25:37.322529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.322469 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e"} err="failed to get container status \"bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e\": rpc error: code = NotFound desc = could not find container \"bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e\": container with ID starting with bf3c7b6bea7a1c68b854a6f5a75b5bcc1c1de0f338ac386d0606094d4b6db01e not found: ID does not exist" Nov 03 07:25:37.322529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.322485 4409 scope.go:122] "RemoveContainer" containerID="58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6" Nov 03 07:25:37.331739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.331634 4409 scope.go:122] "RemoveContainer" containerID="58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6" Nov 03 07:25:37.331946 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:25:37.331928 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6\": container with ID starting with 58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6 not found: ID does not exist" containerID="58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6" Nov 03 07:25:37.331998 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.331953 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6"} err="failed to get container status \"58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6\": rpc error: code = NotFound desc = could not find container \"58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6\": container with ID starting with 58443aef6bd1e23585dfcf56d9f3781382e1ecce0ace9082c4560f8647fe62a6 not found: ID does not exist" Nov 03 07:25:37.423590 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:25:37.423560 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536748c7_12c7_4188_b0e4_0df4ee35a7ab.slice/crio-71dc1efd6bbabe75d63520d2846e8daa91b971b0db58edcb50cdf365399af78e WatchSource:0}: Error finding container 71dc1efd6bbabe75d63520d2846e8daa91b971b0db58edcb50cdf365399af78e: Status 404 returned error can't find the container with id 71dc1efd6bbabe75d63520d2846e8daa91b971b0db58edcb50cdf365399af78e Nov 03 07:25:37.423784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.423725 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl"] Nov 03 07:25:37.426179 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.426147 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:25:37.426267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.426207 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:25:37.426267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.426236 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:25:37.635463 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.635427 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8"] Nov 03 07:25:37.639610 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.639583 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5c169f6a-6769b6f497-ndtt8"] Nov 03 07:25:37.748366 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.748288 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t"] Nov 03 07:25:37.755134 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:37.755107 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5c169f6a-65c4fc5847-2962t"] Nov 03 07:25:38.319500 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.319463 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" event={"ID":"536748c7-12c7-4188-b0e4-0df4ee35a7ab","Type":"ContainerStarted","Data":"f2a640fc9d5663f7b395411c716c3ae5662a0ecd9ca93bd781fff9161639efab"} Nov 03 07:25:38.320003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.319520 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" event={"ID":"536748c7-12c7-4188-b0e4-0df4ee35a7ab","Type":"ContainerStarted","Data":"71dc1efd6bbabe75d63520d2846e8daa91b971b0db58edcb50cdf365399af78e"} Nov 03 07:25:38.443744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.443681 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" podStartSLOduration=2.443666634 podStartE2EDuration="2.443666634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:25:38.440631946 +0000 UTC m=+1593.633173696" watchObservedRunningTime="2026-11-03 07:25:38.443666634 +0000 UTC m=+1593.636208383" Nov 03 07:25:38.697739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.697008 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:25:38.768154 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.768122 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-socket\") pod \"94f95955-229f-446c-ac9f-2fab451a3889\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " Nov 03 07:25:38.768342 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.768175 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-kube-api-access-gtkqx\" (UniqueName: \"kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-kube-api-access-gtkqx\") pod \"94f95955-229f-446c-ac9f-2fab451a3889\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " Nov 03 07:25:38.768342 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.768224 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-envoy\") pod \"94f95955-229f-446c-ac9f-2fab451a3889\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " Nov 03 07:25:38.768342 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.768270 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-data\") pod \"94f95955-229f-446c-ac9f-2fab451a3889\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " Nov 03 07:25:38.768532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.768345 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-istio-token\" (UniqueName: \"kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-istio-token\") pod \"94f95955-229f-446c-ac9f-2fab451a3889\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " Nov 03 07:25:38.768532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.768454 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-socket" pod "94f95955-229f-446c-ac9f-2fab451a3889" (UID: "94f95955-229f-446c-ac9f-2fab451a3889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:25:38.768532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.768480 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-credential-socket\") pod \"94f95955-229f-446c-ac9f-2fab451a3889\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " Nov 03 07:25:38.768687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.768530 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94f95955-229f-446c-ac9f-2fab451a3889-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f95955-229f-446c-ac9f-2fab451a3889-istio-podinfo\") pod \"94f95955-229f-446c-ac9f-2fab451a3889\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " Nov 03 07:25:38.768687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.768584 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f95955-229f-446c-ac9f-2fab451a3889-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f95955-229f-446c-ac9f-2fab451a3889-istiod-ca-cert\") pod \"94f95955-229f-446c-ac9f-2fab451a3889\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " Nov 03 07:25:38.768687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.768636 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-data" pod "94f95955-229f-446c-ac9f-2fab451a3889" (UID: "94f95955-229f-446c-ac9f-2fab451a3889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:25:38.768882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.768811 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-certs\") pod \"94f95955-229f-446c-ac9f-2fab451a3889\" (UID: \"94f95955-229f-446c-ac9f-2fab451a3889\") " Nov 03 07:25:38.769103 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.769079 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-certs" pod "94f95955-229f-446c-ac9f-2fab451a3889" (UID: "94f95955-229f-446c-ac9f-2fab451a3889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:25:38.769203 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.769138 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-credential-socket" pod "94f95955-229f-446c-ac9f-2fab451a3889" (UID: "94f95955-229f-446c-ac9f-2fab451a3889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:25:38.769203 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.769138 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f95955-229f-446c-ac9f-2fab451a3889-istiod-ca-cert" pod "94f95955-229f-446c-ac9f-2fab451a3889" (UID: "94f95955-229f-446c-ac9f-2fab451a3889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:25:38.769734 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.769576 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:38.769734 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.769599 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f95955-229f-446c-ac9f-2fab451a3889-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:38.769734 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.769608 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:38.769734 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.769619 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:38.769734 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.769633 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:38.771791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.771495 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-envoy" pod "94f95955-229f-446c-ac9f-2fab451a3889" (UID: "94f95955-229f-446c-ac9f-2fab451a3889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:25:38.771791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.771726 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-istio-token" pod "94f95955-229f-446c-ac9f-2fab451a3889" (UID: "94f95955-229f-446c-ac9f-2fab451a3889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:25:38.772215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.772186 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-kube-api-access-gtkqx" pod "94f95955-229f-446c-ac9f-2fab451a3889" (UID: "94f95955-229f-446c-ac9f-2fab451a3889"). InnerVolumeSpecName "kube-api-access-gtkqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:25:38.772311 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.772233 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94f95955-229f-446c-ac9f-2fab451a3889-istio-podinfo" pod "94f95955-229f-446c-ac9f-2fab451a3889" (UID: "94f95955-229f-446c-ac9f-2fab451a3889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:25:38.780511 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:25:38.780479 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:25:38.871659 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.871563 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtkqx\" (UniqueName: \"kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-kube-api-access-gtkqx\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:38.871659 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.871611 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f95955-229f-446c-ac9f-2fab451a3889-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:38.871659 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.871629 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f95955-229f-446c-ac9f-2fab451a3889-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:38.871659 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.871643 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f95955-229f-446c-ac9f-2fab451a3889-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:25:38.914492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.914452 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027e5dc0-f742-4388-a8dd-62de2f7651d2" path="/var/lib/kubelet/pods/027e5dc0-f742-4388-a8dd-62de2f7651d2/volumes" Nov 03 07:25:38.915024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.915007 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d42febe-a8a4-4b3c-b449-e19b9c957409" path="/var/lib/kubelet/pods/2d42febe-a8a4-4b3c-b449-e19b9c957409/volumes" Nov 03 07:25:38.946241 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.946214 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:38.952838 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:38.952811 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:39.334233 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.334201 4409 generic.go:356] "Generic (PLEG): container finished" podID="94f95955-229f-446c-ac9f-2fab451a3889" containerID="68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4" exitCode=0 Nov 03 07:25:39.334699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.334263 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" Nov 03 07:25:39.334699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.334284 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" event={"ID":"94f95955-229f-446c-ac9f-2fab451a3889","Type":"ContainerDied","Data":"68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4"} Nov 03 07:25:39.334699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.334349 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd" event={"ID":"94f95955-229f-446c-ac9f-2fab451a3889","Type":"ContainerDied","Data":"fd36e457b1dcdd56bbaeb6158361296e2e729570de383b74816146fe5ce345f9"} Nov 03 07:25:39.334699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.334378 4409 scope.go:122] "RemoveContainer" containerID="68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4" Nov 03 07:25:39.335017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.334852 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:39.336026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.336008 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:25:39.345631 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.345616 4409 scope.go:122] "RemoveContainer" containerID="5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c" Nov 03 07:25:39.355527 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.355506 4409 scope.go:122] "RemoveContainer" containerID="68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4" Nov 03 07:25:39.355891 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:25:39.355866 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4\": container with ID starting with 68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4 not found: ID does not exist" containerID="68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4" Nov 03 07:25:39.355983 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.355899 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4"} err="failed to get container status \"68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4\": rpc error: code = NotFound desc = could not find container \"68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4\": container with ID starting with 68344f6ad15526c5c5801d68daaef4a3c54449c632eec5c639fc7a543e4e2eb4 not found: ID does not exist" Nov 03 07:25:39.355983 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.355916 4409 scope.go:122] "RemoveContainer" containerID="5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c" Nov 03 07:25:39.356229 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:25:39.356205 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c\": container with ID starting with 5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c not found: ID does not exist" containerID="5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c" Nov 03 07:25:39.356276 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.356228 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c"} err="failed to get container status \"5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c\": rpc error: code = NotFound desc = could not find container \"5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c\": container with ID starting with 5b1a316ba463d5fbaf60fcc1d9156542a853b3aabd0357327e872dd46a8e6b7c not found: ID does not exist" Nov 03 07:25:39.644840 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.644757 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd"] Nov 03 07:25:39.648321 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:39.648298 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-67db4bc0-openshift-default-6b845d74fc-stxvd"] Nov 03 07:25:40.915049 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:40.915016 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f95955-229f-446c-ac9f-2fab451a3889" path="/var/lib/kubelet/pods/94f95955-229f-446c-ac9f-2fab451a3889/volumes" Nov 03 07:25:42.299597 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.299562 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx"] Nov 03 07:25:42.305567 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.301217 4409 cpu_manager.go:414] "containerMap: removing container" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" Nov 03 07:25:42.305954 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.305927 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" Nov 03 07:25:42.306080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.306068 4409 cpu_manager.go:414] "containerMap: removing container" podUID="2d42febe-a8a4-4b3c-b449-e19b9c957409" containerName="payload-pre-processing" Nov 03 07:25:42.306172 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.306160 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d42febe-a8a4-4b3c-b449-e19b9c957409" containerName="payload-pre-processing" Nov 03 07:25:42.306406 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.306392 4409 cpu_manager.go:414] "containerMap: removing container" podUID="027e5dc0-f742-4388-a8dd-62de2f7651d2" containerName="payload-processing" Nov 03 07:25:42.306504 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.306493 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027e5dc0-f742-4388-a8dd-62de2f7651d2" containerName="payload-processing" Nov 03 07:25:42.306577 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.306568 4409 cpu_manager.go:414] "containerMap: removing container" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" Nov 03 07:25:42.306657 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.306647 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" Nov 03 07:25:42.306978 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.306956 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" Nov 03 07:25:42.307111 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.307097 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d42febe-a8a4-4b3c-b449-e19b9c957409" containerName="payload-pre-processing" Nov 03 07:25:42.307252 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.307239 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="027e5dc0-f742-4388-a8dd-62de2f7651d2" containerName="payload-processing" Nov 03 07:25:42.334777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.327050 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" Nov 03 07:25:42.422750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.422125 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6d089af5-dockercfg-fpx6b\"" Nov 03 07:25:42.422750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.422227 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6d089af5\"" Nov 03 07:25:42.456739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.436994 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx"] Nov 03 07:25:42.487740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.472189 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d42500-cb20-4baf-bb92-c5cd07b314fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx\" (UID: \"c5d42500-cb20-4baf-bb92-c5cd07b314fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" Nov 03 07:25:42.487740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.472376 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8bqt\" (UniqueName: \"kubernetes.io/projected/c5d42500-cb20-4baf-bb92-c5cd07b314fc-kube-api-access-b8bqt\") pod \"payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx\" (UID: \"c5d42500-cb20-4baf-bb92-c5cd07b314fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" Nov 03 07:25:42.536738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.536669 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd"] Nov 03 07:25:42.556786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.538046 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f95955-229f-446c-ac9f-2fab451a3889" containerName="istio-proxy" Nov 03 07:25:42.556786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.551163 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" Nov 03 07:25:42.575829 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.575113 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d42500-cb20-4baf-bb92-c5cd07b314fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx\" (UID: \"c5d42500-cb20-4baf-bb92-c5cd07b314fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" Nov 03 07:25:42.575829 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.575296 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8bqt\" (UniqueName: \"kubernetes.io/projected/c5d42500-cb20-4baf-bb92-c5cd07b314fc-kube-api-access-b8bqt\") pod \"payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx\" (UID: \"c5d42500-cb20-4baf-bb92-c5cd07b314fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" Nov 03 07:25:42.577604 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.576906 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d42500-cb20-4baf-bb92-c5cd07b314fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx\" (UID: \"c5d42500-cb20-4baf-bb92-c5cd07b314fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" Nov 03 07:25:42.714741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.706200 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwvx\" (UniqueName: \"kubernetes.io/projected/1e21ba32-6131-48a0-b25f-01ba66c688d8-kube-api-access-7rwvx\") pod \"payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd\" (UID: \"1e21ba32-6131-48a0-b25f-01ba66c688d8\") " pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" Nov 03 07:25:42.714741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.706302 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e21ba32-6131-48a0-b25f-01ba66c688d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd\" (UID: \"1e21ba32-6131-48a0-b25f-01ba66c688d8\") " pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" Nov 03 07:25:42.753791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.753039 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8bqt\" (UniqueName: \"kubernetes.io/projected/c5d42500-cb20-4baf-bb92-c5cd07b314fc-kube-api-access-b8bqt\") pod \"payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx\" (UID: \"c5d42500-cb20-4baf-bb92-c5cd07b314fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" Nov 03 07:25:42.753791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.753547 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" Nov 03 07:25:42.809387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.809292 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwvx\" (UniqueName: \"kubernetes.io/projected/1e21ba32-6131-48a0-b25f-01ba66c688d8-kube-api-access-7rwvx\") pod \"payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd\" (UID: \"1e21ba32-6131-48a0-b25f-01ba66c688d8\") " pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" Nov 03 07:25:42.809560 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.809395 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e21ba32-6131-48a0-b25f-01ba66c688d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd\" (UID: \"1e21ba32-6131-48a0-b25f-01ba66c688d8\") " pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" Nov 03 07:25:42.869653 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.822644 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e21ba32-6131-48a0-b25f-01ba66c688d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd\" (UID: \"1e21ba32-6131-48a0-b25f-01ba66c688d8\") " pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" Nov 03 07:25:42.919385 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.919352 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwvx\" (UniqueName: \"kubernetes.io/projected/1e21ba32-6131-48a0-b25f-01ba66c688d8-kube-api-access-7rwvx\") pod \"payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd\" (UID: \"1e21ba32-6131-48a0-b25f-01ba66c688d8\") " pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" Nov 03 07:25:42.921140 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.921110 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd"] Nov 03 07:25:42.924858 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:42.924833 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" Nov 03 07:25:43.363630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:43.363598 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd"] Nov 03 07:25:43.364504 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:25:43.364477 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21ba32_6131_48a0_b25f_01ba66c688d8.slice/crio-7a9b5ad4ef75b73aeb3de712e257b8a9e200e097e97e1fdb89e4e8bebe73726a WatchSource:0}: Error finding container 7a9b5ad4ef75b73aeb3de712e257b8a9e200e097e97e1fdb89e4e8bebe73726a: Status 404 returned error can't find the container with id 7a9b5ad4ef75b73aeb3de712e257b8a9e200e097e97e1fdb89e4e8bebe73726a Nov 03 07:25:43.399798 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:43.399764 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" event={"ID":"1e21ba32-6131-48a0-b25f-01ba66c688d8","Type":"ContainerStarted","Data":"7a9b5ad4ef75b73aeb3de712e257b8a9e200e097e97e1fdb89e4e8bebe73726a"} Nov 03 07:25:43.414279 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:43.414253 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx"] Nov 03 07:25:43.414859 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:25:43.414831 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d42500_cb20_4baf_bb92_c5cd07b314fc.slice/crio-90e6f3f178c228817cdc88fde24277370821a7eb94fd02abaadc4c403791d55f WatchSource:0}: Error finding container 90e6f3f178c228817cdc88fde24277370821a7eb94fd02abaadc4c403791d55f: Status 404 returned error can't find the container with id 90e6f3f178c228817cdc88fde24277370821a7eb94fd02abaadc4c403791d55f Nov 03 07:25:44.408647 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:44.408612 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" event={"ID":"c5d42500-cb20-4baf-bb92-c5cd07b314fc","Type":"ContainerStarted","Data":"b5db4f9351d720be0c92441ebbaee0a556a09f7a18e0dd014c48d960ca5145f2"} Nov 03 07:25:44.409170 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:44.408663 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" event={"ID":"c5d42500-cb20-4baf-bb92-c5cd07b314fc","Type":"ContainerStarted","Data":"90e6f3f178c228817cdc88fde24277370821a7eb94fd02abaadc4c403791d55f"} Nov 03 07:25:44.409170 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:44.408753 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" Nov 03 07:25:44.410516 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:44.410489 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" event={"ID":"1e21ba32-6131-48a0-b25f-01ba66c688d8","Type":"ContainerStarted","Data":"8decf484f6c932dfc46b3ec5021e8886c6f47e51ecc170a32e7492598bd2781f"} Nov 03 07:25:44.410649 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:44.410553 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" Nov 03 07:25:44.529925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:44.529874 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" podStartSLOduration=2.529860485 podStartE2EDuration="2.529860485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:25:44.528681572 +0000 UTC m=+1599.721223322" watchObservedRunningTime="2026-11-03 07:25:44.529860485 +0000 UTC m=+1599.722402234" Nov 03 07:25:44.680894 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:44.678794 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" podStartSLOduration=2.678770524 podStartE2EDuration="2.678770524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:25:44.652522388 +0000 UTC m=+1599.845064140" watchObservedRunningTime="2026-11-03 07:25:44.678770524 +0000 UTC m=+1599.871312316" Nov 03 07:25:47.458784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.451018 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5"] Nov 03 07:25:47.515943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.511935 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5"] Nov 03 07:25:47.515943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.512083 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" Nov 03 07:25:47.598998 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.598964 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-209c4c70\"" Nov 03 07:25:47.658771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.647768 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8psbl\" (UniqueName: \"kubernetes.io/projected/f6755400-98b3-45ce-9943-ebf626e316fb-kube-api-access-8psbl\") pod \"payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5\" (UID: \"f6755400-98b3-45ce-9943-ebf626e316fb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" Nov 03 07:25:47.669932 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.662077 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6755400-98b3-45ce-9943-ebf626e316fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5\" (UID: \"f6755400-98b3-45ce-9943-ebf626e316fb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" Nov 03 07:25:47.669932 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.657584 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-209c4c70-dockercfg-h988h\"" Nov 03 07:25:47.768266 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.768136 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8psbl\" (UniqueName: \"kubernetes.io/projected/f6755400-98b3-45ce-9943-ebf626e316fb-kube-api-access-8psbl\") pod \"payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5\" (UID: \"f6755400-98b3-45ce-9943-ebf626e316fb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" Nov 03 07:25:47.786386 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.786024 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6755400-98b3-45ce-9943-ebf626e316fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5\" (UID: \"f6755400-98b3-45ce-9943-ebf626e316fb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" Nov 03 07:25:47.841871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.841838 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6755400-98b3-45ce-9943-ebf626e316fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5\" (UID: \"f6755400-98b3-45ce-9943-ebf626e316fb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" Nov 03 07:25:47.874763 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.867339 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk"] Nov 03 07:25:47.924659 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.924622 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk"] Nov 03 07:25:47.925024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:47.925004 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" Nov 03 07:25:48.002110 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:48.001087 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911d39d4-dee5-46dd-867d-548ae928ee3c-plugins-config-volume\") pod \"payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk\" (UID: \"911d39d4-dee5-46dd-867d-548ae928ee3c\") " pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" Nov 03 07:25:48.002110 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:48.001287 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sm6t\" (UniqueName: \"kubernetes.io/projected/911d39d4-dee5-46dd-867d-548ae928ee3c-kube-api-access-8sm6t\") pod \"payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk\" (UID: \"911d39d4-dee5-46dd-867d-548ae928ee3c\") " pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" Nov 03 07:25:48.055767 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:48.054468 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8psbl\" (UniqueName: \"kubernetes.io/projected/f6755400-98b3-45ce-9943-ebf626e316fb-kube-api-access-8psbl\") pod \"payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5\" (UID: \"f6755400-98b3-45ce-9943-ebf626e316fb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" Nov 03 07:25:48.103081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:48.103042 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sm6t\" (UniqueName: \"kubernetes.io/projected/911d39d4-dee5-46dd-867d-548ae928ee3c-kube-api-access-8sm6t\") pod \"payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk\" (UID: \"911d39d4-dee5-46dd-867d-548ae928ee3c\") " pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" Nov 03 07:25:48.103242 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:48.103212 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911d39d4-dee5-46dd-867d-548ae928ee3c-plugins-config-volume\") pod \"payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk\" (UID: \"911d39d4-dee5-46dd-867d-548ae928ee3c\") " pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" Nov 03 07:25:48.103963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:48.103941 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911d39d4-dee5-46dd-867d-548ae928ee3c-plugins-config-volume\") pod \"payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk\" (UID: \"911d39d4-dee5-46dd-867d-548ae928ee3c\") " pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" Nov 03 07:25:48.168693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:48.168663 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sm6t\" (UniqueName: \"kubernetes.io/projected/911d39d4-dee5-46dd-867d-548ae928ee3c-kube-api-access-8sm6t\") pod \"payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk\" (UID: \"911d39d4-dee5-46dd-867d-548ae928ee3c\") " pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" Nov 03 07:25:48.329535 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:48.329437 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" Nov 03 07:25:48.336634 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:48.336563 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" Nov 03 07:25:48.768738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:48.765590 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk"] Nov 03 07:25:48.768738 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:25:48.765997 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911d39d4_dee5_46dd_867d_548ae928ee3c.slice/crio-2b52d9150ea8ff9684fc7995922cd8138727191e41ad7a1506eab3669c86387a WatchSource:0}: Error finding container 2b52d9150ea8ff9684fc7995922cd8138727191e41ad7a1506eab3669c86387a: Status 404 returned error can't find the container with id 2b52d9150ea8ff9684fc7995922cd8138727191e41ad7a1506eab3669c86387a Nov 03 07:25:48.839184 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:48.839149 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5"] Nov 03 07:25:48.863368 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:25:48.863339 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6755400_98b3_45ce_9943_ebf626e316fb.slice/crio-ac77179f0af4e01ea262e942c018460bbe8edded6a38a19f8ad0059c3adce415 WatchSource:0}: Error finding container ac77179f0af4e01ea262e942c018460bbe8edded6a38a19f8ad0059c3adce415: Status 404 returned error can't find the container with id ac77179f0af4e01ea262e942c018460bbe8edded6a38a19f8ad0059c3adce415 Nov 03 07:25:49.535738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:49.535642 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" event={"ID":"911d39d4-dee5-46dd-867d-548ae928ee3c","Type":"ContainerStarted","Data":"cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc"} Nov 03 07:25:49.535738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:49.535730 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" event={"ID":"911d39d4-dee5-46dd-867d-548ae928ee3c","Type":"ContainerStarted","Data":"2b52d9150ea8ff9684fc7995922cd8138727191e41ad7a1506eab3669c86387a"} Nov 03 07:25:49.539736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:49.536843 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" Nov 03 07:25:49.539736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:49.539069 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" event={"ID":"f6755400-98b3-45ce-9943-ebf626e316fb","Type":"ContainerStarted","Data":"2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b"} Nov 03 07:25:49.539736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:49.539121 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" event={"ID":"f6755400-98b3-45ce-9943-ebf626e316fb","Type":"ContainerStarted","Data":"ac77179f0af4e01ea262e942c018460bbe8edded6a38a19f8ad0059c3adce415"} Nov 03 07:25:49.540029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:49.539829 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" Nov 03 07:25:49.662801 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:49.662756 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" podStartSLOduration=2.66274247 podStartE2EDuration="2.66274247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:25:49.659653729 +0000 UTC m=+1604.852195475" watchObservedRunningTime="2026-11-03 07:25:49.66274247 +0000 UTC m=+1604.855284217" Nov 03 07:25:49.794791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:49.793870 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" podStartSLOduration=2.793848931 podStartE2EDuration="2.793848931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:25:49.784190939 +0000 UTC m=+1604.976732690" watchObservedRunningTime="2026-11-03 07:25:49.793848931 +0000 UTC m=+1604.986390682" Nov 03 07:25:55.421359 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:55.421326 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" Nov 03 07:25:55.421770 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:25:55.421399 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" Nov 03 07:26:00.310566 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:00.310534 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5"] Nov 03 07:26:00.310984 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:00.310783 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" podUID="8c42291a-6e83-445f-86f7-1af9e7307338" containerName="maas-api" containerID="cri-o://b9c9dc5779bef400651a43d658fda0d98cd1696b1f4459e266f1a2e4f04ee668" gracePeriod=30 Nov 03 07:26:00.742740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:00.738981 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" Nov 03 07:26:00.880149 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:00.879956 4409 generic.go:356] "Generic (PLEG): container finished" podID="8c42291a-6e83-445f-86f7-1af9e7307338" containerID="b9c9dc5779bef400651a43d658fda0d98cd1696b1f4459e266f1a2e4f04ee668" exitCode=0 Nov 03 07:26:00.880149 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:00.880042 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" event={"ID":"8c42291a-6e83-445f-86f7-1af9e7307338","Type":"ContainerDied","Data":"b9c9dc5779bef400651a43d658fda0d98cd1696b1f4459e266f1a2e4f04ee668"} Nov 03 07:26:00.880149 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:00.880094 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" event={"ID":"8c42291a-6e83-445f-86f7-1af9e7307338","Type":"ContainerDied","Data":"0abef7362bc9b1f75e334e8b44376680e508c86f4f444fcdcb372e7e85a437ed"} Nov 03 07:26:00.880149 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:00.880122 4409 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0abef7362bc9b1f75e334e8b44376680e508c86f4f444fcdcb372e7e85a437ed" Nov 03 07:26:00.988784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:00.984568 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:26:01.163946 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.163911 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-maas-api-tls\") pod \"8c42291a-6e83-445f-86f7-1af9e7307338\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " Nov 03 07:26:01.164144 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.164008 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-metrics-certs\") pod \"8c42291a-6e83-445f-86f7-1af9e7307338\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " Nov 03 07:26:01.164144 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.164067 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c42291a-6e83-445f-86f7-1af9e7307338-kube-api-access-zvrlp\" (UniqueName: \"kubernetes.io/projected/8c42291a-6e83-445f-86f7-1af9e7307338-kube-api-access-zvrlp\") pod \"8c42291a-6e83-445f-86f7-1af9e7307338\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " Nov 03 07:26:01.164144 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.164120 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c42291a-6e83-445f-86f7-1af9e7307338-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c42291a-6e83-445f-86f7-1af9e7307338-openshift-service-ca\") pod \"8c42291a-6e83-445f-86f7-1af9e7307338\" (UID: \"8c42291a-6e83-445f-86f7-1af9e7307338\") " Nov 03 07:26:01.164995 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.164966 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c42291a-6e83-445f-86f7-1af9e7307338-openshift-service-ca" pod "8c42291a-6e83-445f-86f7-1af9e7307338" (UID: "8c42291a-6e83-445f-86f7-1af9e7307338"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:26:01.166318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.166291 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-maas-api-tls" pod "8c42291a-6e83-445f-86f7-1af9e7307338" (UID: "8c42291a-6e83-445f-86f7-1af9e7307338"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:26:01.166849 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.166815 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-metrics-certs" pod "8c42291a-6e83-445f-86f7-1af9e7307338" (UID: "8c42291a-6e83-445f-86f7-1af9e7307338"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:26:01.166949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.166900 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c42291a-6e83-445f-86f7-1af9e7307338-kube-api-access-zvrlp" pod "8c42291a-6e83-445f-86f7-1af9e7307338" (UID: "8c42291a-6e83-445f-86f7-1af9e7307338"). InnerVolumeSpecName "kube-api-access-zvrlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:26:01.267254 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.267154 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:01.267464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.267402 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c42291a-6e83-445f-86f7-1af9e7307338-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:01.267464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.267427 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvrlp\" (UniqueName: \"kubernetes.io/projected/8c42291a-6e83-445f-86f7-1af9e7307338-kube-api-access-zvrlp\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:01.267464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.267446 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c42291a-6e83-445f-86f7-1af9e7307338-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:01.565743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.565693 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" Nov 03 07:26:01.886481 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:01.886402 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5" Nov 03 07:26:02.056431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:02.056402 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5"] Nov 03 07:26:02.064112 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:02.064084 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5c169f6a-5ccb99bc6d-mv4p5"] Nov 03 07:26:02.914181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:02.914153 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c42291a-6e83-445f-86f7-1af9e7307338" path="/var/lib/kubelet/pods/8c42291a-6e83-445f-86f7-1af9e7307338/volumes" Nov 03 07:26:04.099746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:04.099697 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps"] Nov 03 07:26:04.100180 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:04.099913 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" containerID="cri-o://af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124" gracePeriod=30 Nov 03 07:26:08.071427 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:08.071391 4409 patch_prober.go:29] interesting pod/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:26:08.071857 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:08.071444 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:26:08.775163 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:26:08.775129 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:26:09.390922 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.390895 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:26:09.560751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.560708 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-data\") pod \"80f64254-ffd6-4900-94fa-c6386145579a\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " Nov 03 07:26:09.560903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.560774 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80f64254-ffd6-4900-94fa-c6386145579a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f64254-ffd6-4900-94fa-c6386145579a-istio-podinfo\") pod \"80f64254-ffd6-4900-94fa-c6386145579a\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " Nov 03 07:26:09.560903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.560811 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-envoy\") pod \"80f64254-ffd6-4900-94fa-c6386145579a\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " Nov 03 07:26:09.560903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.560852 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-socket\") pod \"80f64254-ffd6-4900-94fa-c6386145579a\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " Nov 03 07:26:09.561095 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.560920 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-istio-token\" (UniqueName: \"kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-istio-token\") pod \"80f64254-ffd6-4900-94fa-c6386145579a\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " Nov 03 07:26:09.561095 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.560951 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-certs\") pod \"80f64254-ffd6-4900-94fa-c6386145579a\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " Nov 03 07:26:09.561095 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.561010 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-data" pod "80f64254-ffd6-4900-94fa-c6386145579a" (UID: "80f64254-ffd6-4900-94fa-c6386145579a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:26:09.561095 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.561029 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-kube-api-access-7j7zb\" (UniqueName: \"kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-kube-api-access-7j7zb\") pod \"80f64254-ffd6-4900-94fa-c6386145579a\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " Nov 03 07:26:09.561095 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.561083 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-credential-socket\") pod \"80f64254-ffd6-4900-94fa-c6386145579a\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " Nov 03 07:26:09.561373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.561197 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f64254-ffd6-4900-94fa-c6386145579a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f64254-ffd6-4900-94fa-c6386145579a-istiod-ca-cert\") pod \"80f64254-ffd6-4900-94fa-c6386145579a\" (UID: \"80f64254-ffd6-4900-94fa-c6386145579a\") " Nov 03 07:26:09.561373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.561201 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-socket" pod "80f64254-ffd6-4900-94fa-c6386145579a" (UID: "80f64254-ffd6-4900-94fa-c6386145579a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:26:09.561373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.561237 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-certs" pod "80f64254-ffd6-4900-94fa-c6386145579a" (UID: "80f64254-ffd6-4900-94fa-c6386145579a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:26:09.561514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.561414 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-credential-socket" pod "80f64254-ffd6-4900-94fa-c6386145579a" (UID: "80f64254-ffd6-4900-94fa-c6386145579a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:26:09.561875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.561847 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f64254-ffd6-4900-94fa-c6386145579a-istiod-ca-cert" pod "80f64254-ffd6-4900-94fa-c6386145579a" (UID: "80f64254-ffd6-4900-94fa-c6386145579a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:26:09.562195 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.562172 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f64254-ffd6-4900-94fa-c6386145579a-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:09.562269 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.562203 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:09.562269 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.562217 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:09.562269 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.562231 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:09.562269 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.562245 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:09.563267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.563239 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80f64254-ffd6-4900-94fa-c6386145579a-istio-podinfo" pod "80f64254-ffd6-4900-94fa-c6386145579a" (UID: "80f64254-ffd6-4900-94fa-c6386145579a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:26:09.563267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.563240 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-envoy" pod "80f64254-ffd6-4900-94fa-c6386145579a" (UID: "80f64254-ffd6-4900-94fa-c6386145579a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:26:09.563410 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.563310 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-istio-token" pod "80f64254-ffd6-4900-94fa-c6386145579a" (UID: "80f64254-ffd6-4900-94fa-c6386145579a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:26:09.563755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.563695 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-kube-api-access-7j7zb" pod "80f64254-ffd6-4900-94fa-c6386145579a" (UID: "80f64254-ffd6-4900-94fa-c6386145579a"). InnerVolumeSpecName "kube-api-access-7j7zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:26:09.663874 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.663804 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j7zb\" (UniqueName: \"kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-kube-api-access-7j7zb\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:09.663874 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.663829 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f64254-ffd6-4900-94fa-c6386145579a-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:09.663874 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.663840 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f64254-ffd6-4900-94fa-c6386145579a-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:09.663874 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.663849 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f64254-ffd6-4900-94fa-c6386145579a-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:26:09.945705 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.945635 4409 generic.go:356] "Generic (PLEG): container finished" podID="80f64254-ffd6-4900-94fa-c6386145579a" containerID="af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124" exitCode=0 Nov 03 07:26:09.945705 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.945732 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" Nov 03 07:26:09.945943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.945750 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" event={"ID":"80f64254-ffd6-4900-94fa-c6386145579a","Type":"ContainerDied","Data":"af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124"} Nov 03 07:26:09.945943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.945815 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps" event={"ID":"80f64254-ffd6-4900-94fa-c6386145579a","Type":"ContainerDied","Data":"23107a75228928b955adc18ec37e2c9e1ea580364f6494ad858fbfcde0980043"} Nov 03 07:26:09.945943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.945844 4409 scope.go:122] "RemoveContainer" containerID="af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124" Nov 03 07:26:09.957778 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.957759 4409 scope.go:122] "RemoveContainer" containerID="8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b" Nov 03 07:26:09.967093 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.967075 4409 scope.go:122] "RemoveContainer" containerID="af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124" Nov 03 07:26:09.967380 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:26:09.967348 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124\": container with ID starting with af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124 not found: ID does not exist" containerID="af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124" Nov 03 07:26:09.967444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.967388 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124"} err="failed to get container status \"af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124\": rpc error: code = NotFound desc = could not find container \"af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124\": container with ID starting with af40a854a5f39655b3e194d141dfbdd087edc351ef31bb46af8a53ea2f54e124 not found: ID does not exist" Nov 03 07:26:09.967444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.967406 4409 scope.go:122] "RemoveContainer" containerID="8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b" Nov 03 07:26:09.967707 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:26:09.967673 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b\": container with ID starting with 8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b not found: ID does not exist" containerID="8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b" Nov 03 07:26:09.967707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:09.967729 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b"} err="failed to get container status \"8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b\": rpc error: code = NotFound desc = could not find container \"8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b\": container with ID starting with 8414421ba94a845573dab9d4d1a650b76e848efdc3af4b7a387697734b01343b not found: ID does not exist" Nov 03 07:26:10.124602 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:10.124581 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps"] Nov 03 07:26:10.174648 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:10.174615 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5c169f6a-openshift-default-5d598cd459-lsqps"] Nov 03 07:26:10.913979 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:10.913947 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f64254-ffd6-4900-94fa-c6386145579a" path="/var/lib/kubelet/pods/80f64254-ffd6-4900-94fa-c6386145579a/volumes" Nov 03 07:26:22.633253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.633212 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v"] Nov 03 07:26:22.634130 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.634105 4409 cpu_manager.go:414] "containerMap: removing container" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" Nov 03 07:26:22.634130 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.634129 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" Nov 03 07:26:22.634320 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.634185 4409 cpu_manager.go:414] "containerMap: removing container" podUID="8c42291a-6e83-445f-86f7-1af9e7307338" containerName="maas-api" Nov 03 07:26:22.634320 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.634193 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c42291a-6e83-445f-86f7-1af9e7307338" containerName="maas-api" Nov 03 07:26:22.634421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.634394 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c42291a-6e83-445f-86f7-1af9e7307338" containerName="maas-api" Nov 03 07:26:22.634421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.634417 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" Nov 03 07:26:22.634520 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.634433 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" Nov 03 07:26:22.640095 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.640073 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:22.698840 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.698817 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v"] Nov 03 07:26:22.743576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.743543 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6d089af5-serving-cert\"" Nov 03 07:26:22.746390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.746364 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs2sv\" (UniqueName: \"kubernetes.io/projected/89918fbd-c023-4f3a-9e09-26f620814a79-kube-api-access-zs2sv\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:22.746523 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.746429 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-maas-api-tls\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:22.746523 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.746503 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89918fbd-c023-4f3a-9e09-26f620814a79-openshift-service-ca\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:22.746706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.746690 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-metrics-certs\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:22.847939 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.847907 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89918fbd-c023-4f3a-9e09-26f620814a79-openshift-service-ca\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:22.848100 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.848085 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-metrics-certs\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:22.848152 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.848141 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs2sv\" (UniqueName: \"kubernetes.io/projected/89918fbd-c023-4f3a-9e09-26f620814a79-kube-api-access-zs2sv\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:22.848188 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.848178 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-maas-api-tls\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:22.848317 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:26:22.848305 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-serving-cert: secret "maas-api-e2e-ait-6d089af5-serving-cert" not found Nov 03 07:26:22.848390 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:26:22.848380 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-maas-api-tls podName:89918fbd-c023-4f3a-9e09-26f620814a79 nodeName:}" failed. No retries permitted until 2026-11-03 07:26:23.348363949 +0000 UTC m=+1638.540905693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-maas-api-tls") pod "maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" (UID: "89918fbd-c023-4f3a-9e09-26f620814a79") : secret "maas-api-e2e-ait-6d089af5-serving-cert" not found Nov 03 07:26:22.848523 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.848501 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89918fbd-c023-4f3a-9e09-26f620814a79-openshift-service-ca\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:22.850726 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.850692 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-metrics-certs\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:22.937360 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:22.937282 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs2sv\" (UniqueName: \"kubernetes.io/projected/89918fbd-c023-4f3a-9e09-26f620814a79-kube-api-access-zs2sv\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:23.245496 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.245413 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf"] Nov 03 07:26:23.246325 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.246299 4409 cpu_manager.go:414] "containerMap: removing container" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" Nov 03 07:26:23.246325 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.246325 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f64254-ffd6-4900-94fa-c6386145579a" containerName="istio-proxy" Nov 03 07:26:23.250441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.250413 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.305994 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.305970 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-893d83-openshift-default-dockercfg-fjdl4\"" Nov 03 07:26:23.330306 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.330281 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf"] Nov 03 07:26:23.559898 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.559854 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-credential-socket\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.568740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.560255 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-socket\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.569050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.569017 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-data\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.569221 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.569201 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndrm\" (UniqueName: \"kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-kube-api-access-tndrm\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.569400 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.569384 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istiod-ca-cert\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.586147 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.586109 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-envoy\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.586411 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.586390 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-maas-api-tls\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:23.586703 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.586685 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-podinfo\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.586926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.586884 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-certs\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.587063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.587047 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-token\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.691065 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.691022 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-podinfo\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.700885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.700851 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-certs\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.701162 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.701141 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-token\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.701337 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.701319 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-credential-socket\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.707755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.701642 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-socket\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.707963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.707936 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-socket\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.708052 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.700600 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-maas-api-tls\") pod \"maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:23.708052 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.706520 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-certs\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.708052 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.706595 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-credential-socket\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.708220 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.706378 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-podinfo\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.708323 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.708302 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-data\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.708459 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.708442 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndrm\" (UniqueName: \"kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-kube-api-access-tndrm\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.711861 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.711833 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istiod-ca-cert\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.712158 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.712137 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-envoy\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.727196 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.727162 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-data\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.756739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.728421 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istiod-ca-cert\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.760280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.760247 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-envoy\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.773316 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.773246 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-token\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.839683 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.839567 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndrm\" (UniqueName: \"kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-kube-api-access-tndrm\") pod \"e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:23.854024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.853995 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:23.878282 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:23.878256 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:24.287937 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:24.287148 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v"] Nov 03 07:26:24.460878 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:24.458121 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf"] Nov 03 07:26:24.466269 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:26:24.464239 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc24a90a_dbe9_45f9_bdfc_e09fbfd5b9d6.slice/crio-24532d676d93904f188ce3eb1443b284a030b8f6e729081963a84eb2ff2a8b17 WatchSource:0}: Error finding container 24532d676d93904f188ce3eb1443b284a030b8f6e729081963a84eb2ff2a8b17: Status 404 returned error can't find the container with id 24532d676d93904f188ce3eb1443b284a030b8f6e729081963a84eb2ff2a8b17 Nov 03 07:26:24.498326 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:24.497402 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:26:24.498326 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:24.497498 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:26:24.498326 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:24.497549 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:26:25.229703 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:25.229670 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" event={"ID":"89918fbd-c023-4f3a-9e09-26f620814a79","Type":"ContainerStarted","Data":"e98c23880a2dcbf8f1cc6550067965268bc2c4722a134e9b9fe99bb69de4a751"} Nov 03 07:26:25.232290 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:25.232260 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" event={"ID":"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6","Type":"ContainerStarted","Data":"b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd"} Nov 03 07:26:25.232430 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:25.232326 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" event={"ID":"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6","Type":"ContainerStarted","Data":"24532d676d93904f188ce3eb1443b284a030b8f6e729081963a84eb2ff2a8b17"} Nov 03 07:26:25.359788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:25.359728 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" podStartSLOduration=2.359693736 podStartE2EDuration="2.359693736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:26:25.357096925 +0000 UTC m=+1640.549638676" watchObservedRunningTime="2026-11-03 07:26:25.359693736 +0000 UTC m=+1640.552235488" Nov 03 07:26:25.890077 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:25.890049 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:25.909188 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:25.909157 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:26.309214 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:26.309171 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" event={"ID":"89918fbd-c023-4f3a-9e09-26f620814a79","Type":"ContainerStarted","Data":"884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a"} Nov 03 07:26:26.309899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:26.309874 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:26.310160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:26.310145 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:26.341464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:26.333305 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:26:26.460804 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:26.459524 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" podStartSLOduration=3.516307457 podStartE2EDuration="4.459498891s" totalImagesPullingTime="943.191434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:26:26.444168334 +0000 UTC m=+1641.636710107" watchObservedRunningTime="2026-11-03 07:26:26.459498891 +0000 UTC m=+1641.652040643" Nov 03 07:26:28.620885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.620849 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2"] Nov 03 07:26:28.625239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.625218 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:28.671600 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.671564 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-maas-api-tls\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:28.671845 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.671634 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b0558f-1947-4064-aa94-d936b7f3206c-openshift-service-ca\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:28.671845 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.671770 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-metrics-certs\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:28.671985 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.671845 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l5cg\" (UniqueName: \"kubernetes.io/projected/12b0558f-1947-4064-aa94-d936b7f3206c-kube-api-access-2l5cg\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:28.674565 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.674540 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2"] Nov 03 07:26:28.717884 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.717855 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-209c4c70-serving-cert\"" Nov 03 07:26:28.773299 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.773270 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-maas-api-tls\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:28.773486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.773328 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b0558f-1947-4064-aa94-d936b7f3206c-openshift-service-ca\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:28.773486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.773371 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-metrics-certs\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:28.773486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.773421 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l5cg\" (UniqueName: \"kubernetes.io/projected/12b0558f-1947-4064-aa94-d936b7f3206c-kube-api-access-2l5cg\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:28.773486 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:26:28.773439 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-serving-cert: secret "maas-api-e2e-derive-209c4c70-serving-cert" not found Nov 03 07:26:28.773696 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:26:28.773514 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-maas-api-tls podName:12b0558f-1947-4064-aa94-d936b7f3206c nodeName:}" failed. No retries permitted until 2026-11-03 07:26:29.273490473 +0000 UTC m=+1644.466032222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-maas-api-tls") pod "maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" (UID: "12b0558f-1947-4064-aa94-d936b7f3206c") : secret "maas-api-e2e-derive-209c4c70-serving-cert" not found Nov 03 07:26:28.774100 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.774081 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b0558f-1947-4064-aa94-d936b7f3206c-openshift-service-ca\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:28.776170 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.776151 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-metrics-certs\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:28.839521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:28.839491 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l5cg\" (UniqueName: \"kubernetes.io/projected/12b0558f-1947-4064-aa94-d936b7f3206c-kube-api-access-2l5cg\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:29.296221 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:29.296192 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-maas-api-tls\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:29.298781 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:29.298749 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-maas-api-tls\") pod \"maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:29.539472 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:29.539437 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:29.924556 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:26:29.924527 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b0558f_1947_4064_aa94_d936b7f3206c.slice/crio-2f7dc3f6120d8d8ce8d463c90e807df3b5d635b6c52cfceb60642810c7f45235 WatchSource:0}: Error finding container 2f7dc3f6120d8d8ce8d463c90e807df3b5d635b6c52cfceb60642810c7f45235: Status 404 returned error can't find the container with id 2f7dc3f6120d8d8ce8d463c90e807df3b5d635b6c52cfceb60642810c7f45235 Nov 03 07:26:29.925626 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:29.925601 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2"] Nov 03 07:26:30.372587 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:30.372554 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" event={"ID":"12b0558f-1947-4064-aa94-d936b7f3206c","Type":"ContainerStarted","Data":"2f7dc3f6120d8d8ce8d463c90e807df3b5d635b6c52cfceb60642810c7f45235"} Nov 03 07:26:31.381328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:31.381293 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" event={"ID":"12b0558f-1947-4064-aa94-d936b7f3206c","Type":"ContainerStarted","Data":"3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39"} Nov 03 07:26:31.381787 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:31.381509 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:31.506441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:31.506381 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" podStartSLOduration=2.716772535 podStartE2EDuration="3.506362425s" totalImagesPullingTime="789.58989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:26:31.505159182 +0000 UTC m=+1646.697700972" watchObservedRunningTime="2026-11-03 07:26:31.506362425 +0000 UTC m=+1646.698904178" Nov 03 07:26:32.359127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:32.359102 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:26:33.371313 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.371283 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs"] Nov 03 07:26:33.375614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.375597 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.435693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.435661 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-399d09e5-openshift-default-dockercfg-59gt6\"" Nov 03 07:26:33.490948 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.490917 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs"] Nov 03 07:26:33.579749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.575341 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-credential-socket\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.579749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.575453 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istiod-ca-cert\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.579749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.575524 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-envoy\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.579749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.575645 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-podinfo\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.579749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.575679 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98qnf\" (UniqueName: \"kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-kube-api-access-98qnf\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.579749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.575784 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-certs\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.579749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.575813 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-socket\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.579749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.575946 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-token\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.579749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.576001 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-data\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.791782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.786736 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-credential-socket\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.791782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.786832 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istiod-ca-cert\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.791782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.786900 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-envoy\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.791782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.787011 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-podinfo\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.791782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.787040 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98qnf\" (UniqueName: \"kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-kube-api-access-98qnf\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.791782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.787117 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-certs\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.791782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.787146 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-socket\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.791782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.787266 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-token\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.791782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.787336 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-data\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.791782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.787932 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-data\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.791782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.788190 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-credential-socket\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.803740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.802081 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-certs\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.803740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.802351 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-socket\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.803740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.803228 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istiod-ca-cert\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.819989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.818362 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-envoy\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.832733 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.827856 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-podinfo\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.871737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.870475 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-token\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:33.885819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:33.878494 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98qnf\" (UniqueName: \"kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-kube-api-access-98qnf\") pod \"e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:34.040406 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:34.039279 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:35.299949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:35.299904 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:26:35.311529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:35.300462 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs"] Nov 03 07:26:35.311529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:35.302809 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:26:35.311529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:35.302892 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:26:35.418248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:35.418214 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" event={"ID":"cd9f7361-77a0-4e49-93dc-301cd4c69c40","Type":"ContainerStarted","Data":"fe64403729823c20bfeca987494483e99de5b4ef90f90cddd8ee2b7f4cf0e9de"} Nov 03 07:26:36.462680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:36.462634 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" event={"ID":"cd9f7361-77a0-4e49-93dc-301cd4c69c40","Type":"ContainerStarted","Data":"cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1"} Nov 03 07:26:36.689237 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:36.689170 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" podStartSLOduration=3.689156455 podStartE2EDuration="3.689156455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:26:36.635866783 +0000 UTC m=+1651.828408545" watchObservedRunningTime="2026-11-03 07:26:36.689156455 +0000 UTC m=+1651.881698281" Nov 03 07:26:37.053735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:37.051532 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:37.101740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:37.094199 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:37.393380 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:37.393299 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:26:37.469566 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:37.469533 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:37.470439 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:26:37.470421 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:26:38.777608 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:26:38.777576 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:27:08.353910 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:08.353832 4409 patch_prober.go:29] interesting pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:27:08.354500 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:08.353916 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:27:08.777344 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:27:08.777275 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:27:10.376878 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.376832 4409 patch_prober.go:29] interesting pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:27:10.395616 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.376895 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:27:10.882799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.882769 4409 generic.go:356] "Generic (PLEG): container finished" podID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerID="f2a640fc9d5663f7b395411c716c3ae5662a0ecd9ca93bd781fff9161639efab" exitCode=0 Nov 03 07:27:10.882974 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.882839 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" event={"ID":"536748c7-12c7-4188-b0e4-0df4ee35a7ab","Type":"ContainerDied","Data":"f2a640fc9d5663f7b395411c716c3ae5662a0ecd9ca93bd781fff9161639efab"} Nov 03 07:27:10.883290 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.883273 4409 scope.go:122] "RemoveContainer" containerID="f2a640fc9d5663f7b395411c716c3ae5662a0ecd9ca93bd781fff9161639efab" Nov 03 07:27:10.884544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.884524 4409 generic.go:356] "Generic (PLEG): container finished" podID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerID="77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f" exitCode=0 Nov 03 07:27:10.884622 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.884603 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" event={"ID":"9d107dd0-ba31-4e3c-80fb-377d91700524","Type":"ContainerDied","Data":"77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f"} Nov 03 07:27:10.884902 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.884875 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:27:10.884989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.884948 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:27:10.884989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.884981 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:27:10.885103 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.885014 4409 scope.go:122] "RemoveContainer" containerID="77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f" Nov 03 07:27:10.886425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.886396 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:27:10.886525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.886474 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:27:10.886588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:10.886524 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:27:11.924737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:11.915075 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" event={"ID":"536748c7-12c7-4188-b0e4-0df4ee35a7ab","Type":"ContainerStarted","Data":"c5550701f39a8f14aa7d92fbbb631894233dd631c8fc3923b0cc40261a5d8256"} Nov 03 07:27:11.924737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:11.919314 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" event={"ID":"9d107dd0-ba31-4e3c-80fb-377d91700524","Type":"ContainerStarted","Data":"ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec"} Nov 03 07:27:11.948476 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:11.947100 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:27:11.948476 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:11.947230 4409 patch_prober.go:29] interesting pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 07:27:11.948476 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:11.947288 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 07:27:12.118965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:12.101128 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:27:12.118965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:12.101258 4409 patch_prober.go:29] interesting pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 07:27:12.118965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:12.101307 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 07:27:13.079772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.073190 4409 patch_prober.go:29] interesting pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 07:27:13.079772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.073274 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 07:27:13.090132 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.090099 4409 patch_prober.go:29] interesting pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 07:27:13.097748 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.090361 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 07:27:13.363796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.331804 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw"] Nov 03 07:27:13.686805 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.685927 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw"] Nov 03 07:27:13.686805 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.686071 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" Nov 03 07:27:13.765131 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.764928 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7bzg\" (UniqueName: \"kubernetes.io/projected/436d1679-cf9f-4ec0-a11a-efd0df1b6022-kube-api-access-t7bzg\") pod \"payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw\" (UID: \"436d1679-cf9f-4ec0-a11a-efd0df1b6022\") " pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" Nov 03 07:27:13.765131 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.764977 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436d1679-cf9f-4ec0-a11a-efd0df1b6022-plugins-config-volume\") pod \"payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw\" (UID: \"436d1679-cf9f-4ec0-a11a-efd0df1b6022\") " pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" Nov 03 07:27:13.796739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.790125 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:27:13.814423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.813071 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-295fba7c\"" Nov 03 07:27:13.814423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.813149 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-295fba7c-dockercfg-7n7qf\"" Nov 03 07:27:13.875738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.871247 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7bzg\" (UniqueName: \"kubernetes.io/projected/436d1679-cf9f-4ec0-a11a-efd0df1b6022-kube-api-access-t7bzg\") pod \"payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw\" (UID: \"436d1679-cf9f-4ec0-a11a-efd0df1b6022\") " pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" Nov 03 07:27:13.875738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.871301 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436d1679-cf9f-4ec0-a11a-efd0df1b6022-plugins-config-volume\") pod \"payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw\" (UID: \"436d1679-cf9f-4ec0-a11a-efd0df1b6022\") " pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" Nov 03 07:27:13.895738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.892441 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436d1679-cf9f-4ec0-a11a-efd0df1b6022-plugins-config-volume\") pod \"payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw\" (UID: \"436d1679-cf9f-4ec0-a11a-efd0df1b6022\") " pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" Nov 03 07:27:13.950975 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.947021 4409 patch_prober.go:29] interesting pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 07:27:13.950975 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.947072 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 07:27:13.950975 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:13.950437 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7bzg\" (UniqueName: \"kubernetes.io/projected/436d1679-cf9f-4ec0-a11a-efd0df1b6022-kube-api-access-t7bzg\") pod \"payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw\" (UID: \"436d1679-cf9f-4ec0-a11a-efd0df1b6022\") " pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" Nov 03 07:27:14.088739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:14.087906 4409 patch_prober.go:29] interesting pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 07:27:14.088739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:14.087974 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 07:27:14.127339 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:14.127305 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" Nov 03 07:27:14.628745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:14.621547 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw"] Nov 03 07:27:14.629771 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:27:14.629701 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436d1679_cf9f_4ec0_a11a_efd0df1b6022.slice/crio-b073eb6ab42b3b569a9be99225db9997739136be5762b89248a110354e98a979 WatchSource:0}: Error finding container b073eb6ab42b3b569a9be99225db9997739136be5762b89248a110354e98a979: Status 404 returned error can't find the container with id b073eb6ab42b3b569a9be99225db9997739136be5762b89248a110354e98a979 Nov 03 07:27:14.946931 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:14.946849 4409 patch_prober.go:29] interesting pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 07:27:14.947138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:14.947119 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 07:27:15.095128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:15.095089 4409 patch_prober.go:29] interesting pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 07:27:15.095702 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:15.095669 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 07:27:15.261454 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:15.261370 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" event={"ID":"436d1679-cf9f-4ec0-a11a-efd0df1b6022","Type":"ContainerStarted","Data":"c8d291fddae34ff874b05988035a3610576b5969f6eaf9f47601d43c97691b95"} Nov 03 07:27:15.261600 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:15.261467 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" event={"ID":"436d1679-cf9f-4ec0-a11a-efd0df1b6022","Type":"ContainerStarted","Data":"b073eb6ab42b3b569a9be99225db9997739136be5762b89248a110354e98a979"} Nov 03 07:27:15.261600 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:15.261597 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" Nov 03 07:27:15.408331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:15.408266 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" podStartSLOduration=2.408249389 podStartE2EDuration="2.408249389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:27:15.407505764 +0000 UTC m=+1690.600047515" watchObservedRunningTime="2026-11-03 07:27:15.408249389 +0000 UTC m=+1690.600791140" Nov 03 07:27:15.947744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:15.947168 4409 patch_prober.go:29] interesting pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 07:27:15.947744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:15.947224 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 07:27:16.088263 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:16.088226 4409 patch_prober.go:29] interesting pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 07:27:16.088521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:16.088495 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 07:27:16.946834 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:16.946803 4409 patch_prober.go:29] interesting pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 07:27:16.947293 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:16.946856 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 07:27:17.087607 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:17.087571 4409 patch_prober.go:29] interesting pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 07:27:17.087926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:17.087899 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 07:27:17.947542 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:17.947510 4409 patch_prober.go:29] interesting pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 07:27:17.948014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:17.947583 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 07:27:18.087584 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:18.087544 4409 patch_prober.go:29] interesting pod/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 07:27:18.087782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:18.087628 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 07:27:18.950832 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:18.950804 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:27:18.951228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:18.951212 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:27:18.951968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:18.951946 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:27:19.091851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:19.091824 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:27:19.092125 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:19.092100 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:27:19.092664 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:19.092647 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:27:22.354248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.354218 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4"] Nov 03 07:27:22.358058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.358033 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:22.415004 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.414977 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-893d83-serving-cert\"" Nov 03 07:27:22.419647 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.419621 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4"] Nov 03 07:27:22.427992 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.427965 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlvqw\" (UniqueName: \"kubernetes.io/projected/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-kube-api-access-jlvqw\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:22.428121 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.428049 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:22.428229 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.428199 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:22.428271 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.428257 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:22.529684 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.529649 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlvqw\" (UniqueName: \"kubernetes.io/projected/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-kube-api-access-jlvqw\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:22.529864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.529743 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:22.529864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.529813 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:22.529864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.529847 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:22.530028 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:27:22.529982 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-serving-cert: secret "maas-api-e2e-shared-a-w5-893d83-serving-cert" not found Nov 03 07:27:22.530078 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:27:22.530050 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-maas-api-tls podName:bac1b906-cbe4-4b96-9b01-dd71f6c0dacc nodeName:}" failed. No retries permitted until 2026-11-03 07:27:23.030029482 +0000 UTC m=+1698.222571232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-maas-api-tls") pod "maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" (UID: "bac1b906-cbe4-4b96-9b01-dd71f6c0dacc") : secret "maas-api-e2e-shared-a-w5-893d83-serving-cert" not found Nov 03 07:27:22.530458 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.530434 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:22.532457 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.532434 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:22.629236 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:22.625008 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlvqw\" (UniqueName: \"kubernetes.io/projected/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-kube-api-access-jlvqw\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:23.093024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:23.092991 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:23.095514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:23.095490 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:23.270855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:23.270829 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:23.654860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:23.654833 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4"] Nov 03 07:27:23.655169 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:27:23.654836 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac1b906_cbe4_4b96_9b01_dd71f6c0dacc.slice/crio-7dbfe5ac9a640b1c236574c39114edede68cc5165b21446fc10996645a540405 WatchSource:0}: Error finding container 7dbfe5ac9a640b1c236574c39114edede68cc5165b21446fc10996645a540405: Status 404 returned error can't find the container with id 7dbfe5ac9a640b1c236574c39114edede68cc5165b21446fc10996645a540405 Nov 03 07:27:24.407307 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:24.407273 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" event={"ID":"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc","Type":"ContainerStarted","Data":"7dbfe5ac9a640b1c236574c39114edede68cc5165b21446fc10996645a540405"} Nov 03 07:27:25.415810 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:25.415773 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" event={"ID":"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc","Type":"ContainerStarted","Data":"4da62902c89e1e1fabea1f7cc917133f44f986dfeae65597e68c1a17f333060b"} Nov 03 07:27:25.416204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:25.415884 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:25.547752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:25.547609 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" podStartSLOduration=2.683233017 podStartE2EDuration="3.547594761s" totalImagesPullingTime="864.361744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:27:25.547329666 +0000 UTC m=+1700.739871417" watchObservedRunningTime="2026-11-03 07:27:25.547594761 +0000 UTC m=+1700.740136510" Nov 03 07:27:26.301519 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:26.301493 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" Nov 03 07:27:31.429393 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:31.429359 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:27:38.775934 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:27:38.775897 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:27:39.006389 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.006350 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4"] Nov 03 07:27:39.012185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.012150 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.105626 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.105591 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-metrics-certs\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.105863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.105707 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d24c0370-2c02-4cc9-a456-60059748b1cb-openshift-service-ca\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.105944 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.105901 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j788q\" (UniqueName: \"kubernetes.io/projected/d24c0370-2c02-4cc9-a456-60059748b1cb-kube-api-access-j788q\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.106012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.105953 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-maas-api-tls\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.115284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.115251 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-399d09e5-serving-cert\"" Nov 03 07:27:39.119574 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.119548 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4"] Nov 03 07:27:39.207392 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.207356 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-metrics-certs\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.207567 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.207423 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d24c0370-2c02-4cc9-a456-60059748b1cb-openshift-service-ca\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.207567 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.207552 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j788q\" (UniqueName: \"kubernetes.io/projected/d24c0370-2c02-4cc9-a456-60059748b1cb-kube-api-access-j788q\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.207676 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.207587 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-maas-api-tls\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.207776 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:27:39.207758 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-serving-cert: secret "maas-api-e2e-mt-399d09e5-serving-cert" not found Nov 03 07:27:39.207856 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:27:39.207841 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-maas-api-tls podName:d24c0370-2c02-4cc9-a456-60059748b1cb nodeName:}" failed. No retries permitted until 2026-11-03 07:27:39.707819236 +0000 UTC m=+1714.900360967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-maas-api-tls") pod "maas-api-e2e-mt-399d09e5-546995d-zp6k4" (UID: "d24c0370-2c02-4cc9-a456-60059748b1cb") : secret "maas-api-e2e-mt-399d09e5-serving-cert" not found Nov 03 07:27:39.208268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.208250 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d24c0370-2c02-4cc9-a456-60059748b1cb-openshift-service-ca\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.210117 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.210091 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-metrics-certs\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.320705 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.320573 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j788q\" (UniqueName: \"kubernetes.io/projected/d24c0370-2c02-4cc9-a456-60059748b1cb-kube-api-access-j788q\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.748652 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.748619 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-maas-api-tls\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.751445 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.751410 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-maas-api-tls\") pod \"maas-api-e2e-mt-399d09e5-546995d-zp6k4\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:39.927156 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:39.927121 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:40.321958 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:40.321930 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4"] Nov 03 07:27:40.322103 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:27:40.321934 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24c0370_2c02_4cc9_a456_60059748b1cb.slice/crio-18648f5d1d625931f8384aa0fe7f3295bf69066fe6db7e3a786e8c3c880aa1d5 WatchSource:0}: Error finding container 18648f5d1d625931f8384aa0fe7f3295bf69066fe6db7e3a786e8c3c880aa1d5: Status 404 returned error can't find the container with id 18648f5d1d625931f8384aa0fe7f3295bf69066fe6db7e3a786e8c3c880aa1d5 Nov 03 07:27:40.591107 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:40.591011 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" event={"ID":"d24c0370-2c02-4cc9-a456-60059748b1cb","Type":"ContainerStarted","Data":"18648f5d1d625931f8384aa0fe7f3295bf69066fe6db7e3a786e8c3c880aa1d5"} Nov 03 07:27:41.599978 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:41.599939 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" event={"ID":"d24c0370-2c02-4cc9-a456-60059748b1cb","Type":"ContainerStarted","Data":"b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e"} Nov 03 07:27:41.600423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:41.600047 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:41.722335 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:41.722271 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" podStartSLOduration=2.817935799 podStartE2EDuration="3.722254931s" totalImagesPullingTime="904.319132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:27:41.719164463 +0000 UTC m=+1716.911706214" watchObservedRunningTime="2026-11-03 07:27:41.722254931 +0000 UTC m=+1716.914796680" Nov 03 07:27:47.629160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:47.629133 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:27:48.594936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:48.594898 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd"] Nov 03 07:27:48.696616 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:48.696586 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx"] Nov 03 07:27:51.136304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:51.136267 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk"] Nov 03 07:27:51.256583 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:51.256531 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5"] Nov 03 07:27:53.595808 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:53.595761 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" podUID="1e21ba32-6131-48a0-b25f-01ba66c688d8" containerName="payload-processing" containerID="cri-o://8decf484f6c932dfc46b3ec5021e8886c6f47e51ecc170a32e7492598bd2781f" gracePeriod=25 Nov 03 07:27:53.697697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:53.697647 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" podUID="c5d42500-cb20-4baf-bb92-c5cd07b314fc" containerName="payload-pre-processing" containerID="cri-o://b5db4f9351d720be0c92441ebbaee0a556a09f7a18e0dd014c48d960ca5145f2" gracePeriod=25 Nov 03 07:27:53.827785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:53.827750 4409 generic.go:356] "Generic (PLEG): container finished" podID="c5d42500-cb20-4baf-bb92-c5cd07b314fc" containerID="b5db4f9351d720be0c92441ebbaee0a556a09f7a18e0dd014c48d960ca5145f2" exitCode=0 Nov 03 07:27:53.828005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:53.827868 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" event={"ID":"c5d42500-cb20-4baf-bb92-c5cd07b314fc","Type":"ContainerDied","Data":"b5db4f9351d720be0c92441ebbaee0a556a09f7a18e0dd014c48d960ca5145f2"} Nov 03 07:27:53.829778 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:53.829749 4409 generic.go:356] "Generic (PLEG): container finished" podID="1e21ba32-6131-48a0-b25f-01ba66c688d8" containerID="8decf484f6c932dfc46b3ec5021e8886c6f47e51ecc170a32e7492598bd2781f" exitCode=0 Nov 03 07:27:53.829894 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:53.829798 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" event={"ID":"1e21ba32-6131-48a0-b25f-01ba66c688d8","Type":"ContainerDied","Data":"8decf484f6c932dfc46b3ec5021e8886c6f47e51ecc170a32e7492598bd2781f"} Nov 03 07:27:53.885204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:53.885073 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" Nov 03 07:27:53.999036 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:53.999008 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" Nov 03 07:27:54.002238 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.002212 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e21ba32-6131-48a0-b25f-01ba66c688d8-kube-api-access-7rwvx\" (UniqueName: \"kubernetes.io/projected/1e21ba32-6131-48a0-b25f-01ba66c688d8-kube-api-access-7rwvx\") pod \"1e21ba32-6131-48a0-b25f-01ba66c688d8\" (UID: \"1e21ba32-6131-48a0-b25f-01ba66c688d8\") " Nov 03 07:27:54.002374 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.002356 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e21ba32-6131-48a0-b25f-01ba66c688d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e21ba32-6131-48a0-b25f-01ba66c688d8-plugins-config-volume\") pod \"1e21ba32-6131-48a0-b25f-01ba66c688d8\" (UID: \"1e21ba32-6131-48a0-b25f-01ba66c688d8\") " Nov 03 07:27:54.002680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.002655 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e21ba32-6131-48a0-b25f-01ba66c688d8-plugins-config-volume" pod "1e21ba32-6131-48a0-b25f-01ba66c688d8" (UID: "1e21ba32-6131-48a0-b25f-01ba66c688d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:27:54.003121 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.003096 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e21ba32-6131-48a0-b25f-01ba66c688d8-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:27:54.004760 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.004694 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e21ba32-6131-48a0-b25f-01ba66c688d8-kube-api-access-7rwvx" pod "1e21ba32-6131-48a0-b25f-01ba66c688d8" (UID: "1e21ba32-6131-48a0-b25f-01ba66c688d8"). InnerVolumeSpecName "kube-api-access-7rwvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:27:54.104544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.104460 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d42500-cb20-4baf-bb92-c5cd07b314fc-kube-api-access-b8bqt\" (UniqueName: \"kubernetes.io/projected/c5d42500-cb20-4baf-bb92-c5cd07b314fc-kube-api-access-b8bqt\") pod \"c5d42500-cb20-4baf-bb92-c5cd07b314fc\" (UID: \"c5d42500-cb20-4baf-bb92-c5cd07b314fc\") " Nov 03 07:27:54.104733 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.104661 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d42500-cb20-4baf-bb92-c5cd07b314fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d42500-cb20-4baf-bb92-c5cd07b314fc-plugins-config-volume\") pod \"c5d42500-cb20-4baf-bb92-c5cd07b314fc\" (UID: \"c5d42500-cb20-4baf-bb92-c5cd07b314fc\") " Nov 03 07:27:54.105003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.104981 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d42500-cb20-4baf-bb92-c5cd07b314fc-plugins-config-volume" pod "c5d42500-cb20-4baf-bb92-c5cd07b314fc" (UID: "c5d42500-cb20-4baf-bb92-c5cd07b314fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:27:54.105370 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.105351 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rwvx\" (UniqueName: \"kubernetes.io/projected/1e21ba32-6131-48a0-b25f-01ba66c688d8-kube-api-access-7rwvx\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:27:54.105452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.105375 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d42500-cb20-4baf-bb92-c5cd07b314fc-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:27:54.106699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.106678 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d42500-cb20-4baf-bb92-c5cd07b314fc-kube-api-access-b8bqt" pod "c5d42500-cb20-4baf-bb92-c5cd07b314fc" (UID: "c5d42500-cb20-4baf-bb92-c5cd07b314fc"). InnerVolumeSpecName "kube-api-access-b8bqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:27:54.206935 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.206895 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8bqt\" (UniqueName: \"kubernetes.io/projected/c5d42500-cb20-4baf-bb92-c5cd07b314fc-kube-api-access-b8bqt\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:27:54.838148 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.838119 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" Nov 03 07:27:54.838638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.838152 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx" event={"ID":"c5d42500-cb20-4baf-bb92-c5cd07b314fc","Type":"ContainerDied","Data":"90e6f3f178c228817cdc88fde24277370821a7eb94fd02abaadc4c403791d55f"} Nov 03 07:27:54.838638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.838220 4409 scope.go:122] "RemoveContainer" containerID="b5db4f9351d720be0c92441ebbaee0a556a09f7a18e0dd014c48d960ca5145f2" Nov 03 07:27:54.845647 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.842922 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" event={"ID":"1e21ba32-6131-48a0-b25f-01ba66c688d8","Type":"ContainerDied","Data":"7a9b5ad4ef75b73aeb3de712e257b8a9e200e097e97e1fdb89e4e8bebe73726a"} Nov 03 07:27:54.845647 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.843083 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd" Nov 03 07:27:54.861419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:54.861402 4409 scope.go:122] "RemoveContainer" containerID="8decf484f6c932dfc46b3ec5021e8886c6f47e51ecc170a32e7492598bd2781f" Nov 03 07:27:55.033907 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:55.033877 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx"] Nov 03 07:27:55.045406 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:55.045375 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d089af5-c489f7775-bzclx"] Nov 03 07:27:55.175685 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:55.175598 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd"] Nov 03 07:27:55.179752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:55.179703 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d089af5-6764877fbf-5k7cd"] Nov 03 07:27:56.137289 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.137250 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" podUID="911d39d4-dee5-46dd-867d-548ae928ee3c" containerName="payload-processing" containerID="cri-o://cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc" gracePeriod=25 Nov 03 07:27:56.257360 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.257333 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" podUID="f6755400-98b3-45ce-9943-ebf626e316fb" containerName="payload-pre-processing" containerID="cri-o://2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b" gracePeriod=25 Nov 03 07:27:56.428761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.428678 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" Nov 03 07:27:56.536150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.536127 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" Nov 03 07:27:56.539464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.539441 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911d39d4-dee5-46dd-867d-548ae928ee3c-kube-api-access-8sm6t\" (UniqueName: \"kubernetes.io/projected/911d39d4-dee5-46dd-867d-548ae928ee3c-kube-api-access-8sm6t\") pod \"911d39d4-dee5-46dd-867d-548ae928ee3c\" (UID: \"911d39d4-dee5-46dd-867d-548ae928ee3c\") " Nov 03 07:27:56.539599 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.539583 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911d39d4-dee5-46dd-867d-548ae928ee3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911d39d4-dee5-46dd-867d-548ae928ee3c-plugins-config-volume\") pod \"911d39d4-dee5-46dd-867d-548ae928ee3c\" (UID: \"911d39d4-dee5-46dd-867d-548ae928ee3c\") " Nov 03 07:27:56.539939 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.539917 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911d39d4-dee5-46dd-867d-548ae928ee3c-plugins-config-volume" pod "911d39d4-dee5-46dd-867d-548ae928ee3c" (UID: "911d39d4-dee5-46dd-867d-548ae928ee3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:27:56.540517 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.540499 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911d39d4-dee5-46dd-867d-548ae928ee3c-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:27:56.541797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.541769 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911d39d4-dee5-46dd-867d-548ae928ee3c-kube-api-access-8sm6t" pod "911d39d4-dee5-46dd-867d-548ae928ee3c" (UID: "911d39d4-dee5-46dd-867d-548ae928ee3c"). InnerVolumeSpecName "kube-api-access-8sm6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:27:56.642038 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.641967 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6755400-98b3-45ce-9943-ebf626e316fb-kube-api-access-8psbl\" (UniqueName: \"kubernetes.io/projected/f6755400-98b3-45ce-9943-ebf626e316fb-kube-api-access-8psbl\") pod \"f6755400-98b3-45ce-9943-ebf626e316fb\" (UID: \"f6755400-98b3-45ce-9943-ebf626e316fb\") " Nov 03 07:27:56.642038 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.642022 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6755400-98b3-45ce-9943-ebf626e316fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6755400-98b3-45ce-9943-ebf626e316fb-plugins-config-volume\") pod \"f6755400-98b3-45ce-9943-ebf626e316fb\" (UID: \"f6755400-98b3-45ce-9943-ebf626e316fb\") " Nov 03 07:27:56.642397 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.642366 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6755400-98b3-45ce-9943-ebf626e316fb-plugins-config-volume" pod "f6755400-98b3-45ce-9943-ebf626e316fb" (UID: "f6755400-98b3-45ce-9943-ebf626e316fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:27:56.642604 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.642589 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6755400-98b3-45ce-9943-ebf626e316fb-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:27:56.642649 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.642609 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sm6t\" (UniqueName: \"kubernetes.io/projected/911d39d4-dee5-46dd-867d-548ae928ee3c-kube-api-access-8sm6t\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:27:56.644346 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.644326 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6755400-98b3-45ce-9943-ebf626e316fb-kube-api-access-8psbl" pod "f6755400-98b3-45ce-9943-ebf626e316fb" (UID: "f6755400-98b3-45ce-9943-ebf626e316fb"). InnerVolumeSpecName "kube-api-access-8psbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:27:56.743860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.743830 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8psbl\" (UniqueName: \"kubernetes.io/projected/f6755400-98b3-45ce-9943-ebf626e316fb-kube-api-access-8psbl\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:27:56.861086 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.861057 4409 generic.go:356] "Generic (PLEG): container finished" podID="911d39d4-dee5-46dd-867d-548ae928ee3c" containerID="cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc" exitCode=0 Nov 03 07:27:56.861253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.861105 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" Nov 03 07:27:56.861253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.861147 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" event={"ID":"911d39d4-dee5-46dd-867d-548ae928ee3c","Type":"ContainerDied","Data":"cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc"} Nov 03 07:27:56.861253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.861205 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk" event={"ID":"911d39d4-dee5-46dd-867d-548ae928ee3c","Type":"ContainerDied","Data":"2b52d9150ea8ff9684fc7995922cd8138727191e41ad7a1506eab3669c86387a"} Nov 03 07:27:56.861253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.861232 4409 scope.go:122] "RemoveContainer" containerID="cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc" Nov 03 07:27:56.862953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.862930 4409 generic.go:356] "Generic (PLEG): container finished" podID="f6755400-98b3-45ce-9943-ebf626e316fb" containerID="2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b" exitCode=0 Nov 03 07:27:56.863060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.862963 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" event={"ID":"f6755400-98b3-45ce-9943-ebf626e316fb","Type":"ContainerDied","Data":"2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b"} Nov 03 07:27:56.863060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.862991 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" Nov 03 07:27:56.863060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.863007 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5" event={"ID":"f6755400-98b3-45ce-9943-ebf626e316fb","Type":"ContainerDied","Data":"ac77179f0af4e01ea262e942c018460bbe8edded6a38a19f8ad0059c3adce415"} Nov 03 07:27:56.872588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.872567 4409 scope.go:122] "RemoveContainer" containerID="cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc" Nov 03 07:27:56.872882 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:27:56.872865 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc\": container with ID starting with cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc not found: ID does not exist" containerID="cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc" Nov 03 07:27:56.872941 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.872888 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc"} err="failed to get container status \"cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc\": rpc error: code = NotFound desc = could not find container \"cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc\": container with ID starting with cc726fea23b551cfd1b0a493210bd07b0fb70b8030f95dcee51c66e94df1f8fc not found: ID does not exist" Nov 03 07:27:56.872941 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.872908 4409 scope.go:122] "RemoveContainer" containerID="2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b" Nov 03 07:27:56.882280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.882261 4409 scope.go:122] "RemoveContainer" containerID="2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b" Nov 03 07:27:56.882548 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:27:56.882525 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b\": container with ID starting with 2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b not found: ID does not exist" containerID="2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b" Nov 03 07:27:56.882633 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.882553 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b"} err="failed to get container status \"2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b\": rpc error: code = NotFound desc = could not find container \"2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b\": container with ID starting with 2964dfdb6befd0532bd16ff6241b9ebe5a4a1d006b6accb8a7cfacce739e166b not found: ID does not exist" Nov 03 07:27:56.914401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.914346 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e21ba32-6131-48a0-b25f-01ba66c688d8" path="/var/lib/kubelet/pods/1e21ba32-6131-48a0-b25f-01ba66c688d8/volumes" Nov 03 07:27:56.914892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:56.914874 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d42500-cb20-4baf-bb92-c5cd07b314fc" path="/var/lib/kubelet/pods/c5d42500-cb20-4baf-bb92-c5cd07b314fc/volumes" Nov 03 07:27:57.158347 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:57.152013 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk"] Nov 03 07:27:57.336739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:57.318818 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-209c4c70-547cdcf4dc-g4cjk"] Nov 03 07:27:57.358393 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:57.358357 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5"] Nov 03 07:27:57.365793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:57.365762 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-209c4c70-56c65d7fdc-pfgv5"] Nov 03 07:27:58.917316 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:58.917284 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911d39d4-dee5-46dd-867d-548ae928ee3c" path="/var/lib/kubelet/pods/911d39d4-dee5-46dd-867d-548ae928ee3c/volumes" Nov 03 07:27:58.917848 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:27:58.917798 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6755400-98b3-45ce-9943-ebf626e316fb" path="/var/lib/kubelet/pods/f6755400-98b3-45ce-9943-ebf626e316fb/volumes" Nov 03 07:28:08.328944 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:08.328830 4409 patch_prober.go:29] interesting pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": read tcp 10.133.0.2:34240->10.133.0.104:15021: read: connection reset by peer" start-of-body= Nov 03 07:28:08.329312 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:08.328934 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": read tcp 10.133.0.2:34240->10.133.0.104:15021: read: connection reset by peer" Nov 03 07:28:08.775165 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:28:08.775086 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:28:08.981097 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:08.981070 4409 generic.go:356] "Generic (PLEG): container finished" podID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerID="cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1" exitCode=0 Nov 03 07:28:08.981296 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:08.981147 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" event={"ID":"cd9f7361-77a0-4e49-93dc-301cd4c69c40","Type":"ContainerDied","Data":"cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1"} Nov 03 07:28:08.981529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:08.981514 4409 scope.go:122] "RemoveContainer" containerID="cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1" Nov 03 07:28:08.983150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:08.983119 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:28:08.983232 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:08.983189 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:28:08.983232 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:08.983219 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:28:09.831425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.831393 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr"] Nov 03 07:28:09.832159 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.832145 4409 cpu_manager.go:414] "containerMap: removing container" podUID="1e21ba32-6131-48a0-b25f-01ba66c688d8" containerName="payload-processing" Nov 03 07:28:09.832159 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.832160 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e21ba32-6131-48a0-b25f-01ba66c688d8" containerName="payload-processing" Nov 03 07:28:09.832244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.832176 4409 cpu_manager.go:414] "containerMap: removing container" podUID="c5d42500-cb20-4baf-bb92-c5cd07b314fc" containerName="payload-pre-processing" Nov 03 07:28:09.832244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.832181 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d42500-cb20-4baf-bb92-c5cd07b314fc" containerName="payload-pre-processing" Nov 03 07:28:09.832244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.832204 4409 cpu_manager.go:414] "containerMap: removing container" podUID="911d39d4-dee5-46dd-867d-548ae928ee3c" containerName="payload-processing" Nov 03 07:28:09.832244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.832208 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911d39d4-dee5-46dd-867d-548ae928ee3c" containerName="payload-processing" Nov 03 07:28:09.832244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.832232 4409 cpu_manager.go:414] "containerMap: removing container" podUID="f6755400-98b3-45ce-9943-ebf626e316fb" containerName="payload-pre-processing" Nov 03 07:28:09.832244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.832236 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6755400-98b3-45ce-9943-ebf626e316fb" containerName="payload-pre-processing" Nov 03 07:28:09.832421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.832341 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6755400-98b3-45ce-9943-ebf626e316fb" containerName="payload-pre-processing" Nov 03 07:28:09.832421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.832354 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e21ba32-6131-48a0-b25f-01ba66c688d8" containerName="payload-processing" Nov 03 07:28:09.832421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.832362 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d42500-cb20-4baf-bb92-c5cd07b314fc" containerName="payload-pre-processing" Nov 03 07:28:09.832421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.832370 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="911d39d4-dee5-46dd-867d-548ae928ee3c" containerName="payload-processing" Nov 03 07:28:09.835091 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.835072 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:09.896916 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.896881 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr"] Nov 03 07:28:09.940084 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.940054 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-295fba7c-serving-cert\"" Nov 03 07:28:09.954583 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.954547 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-maas-api-tls\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:09.954740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.954608 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nltbm\" (UniqueName: \"kubernetes.io/projected/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-kube-api-access-nltbm\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:09.954806 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.954767 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-metrics-certs\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:09.954859 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:09.954808 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-openshift-service-ca\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:10.007246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.007199 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" event={"ID":"cd9f7361-77a0-4e49-93dc-301cd4c69c40","Type":"ContainerStarted","Data":"8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb"} Nov 03 07:28:10.040661 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.040629 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:28:10.040818 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.040793 4409 patch_prober.go:29] interesting pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 07:28:10.040863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.040840 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 07:28:10.056740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.056694 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-maas-api-tls\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:10.056851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.056768 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nltbm\" (UniqueName: \"kubernetes.io/projected/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-kube-api-access-nltbm\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:10.056890 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:28:10.056851 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-serving-cert: secret "maas-api-e2e-mt-295fba7c-serving-cert" not found Nov 03 07:28:10.056890 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.056875 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-metrics-certs\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:10.056961 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:28:10.056907 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-maas-api-tls podName:533f3d9f-f1f3-47b1-8bfb-a145524ddd4f nodeName:}" failed. No retries permitted until 2026-11-03 07:28:10.556892661 +0000 UTC m=+1745.749434389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-maas-api-tls") pod "maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" (UID: "533f3d9f-f1f3-47b1-8bfb-a145524ddd4f") : secret "maas-api-e2e-mt-295fba7c-serving-cert" not found Nov 03 07:28:10.056961 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.056938 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-openshift-service-ca\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:10.057509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.057484 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-openshift-service-ca\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:10.059395 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.059369 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-metrics-certs\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:10.152785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.148974 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nltbm\" (UniqueName: \"kubernetes.io/projected/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-kube-api-access-nltbm\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:10.583740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.579620 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-maas-api-tls\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:10.591762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.591730 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-maas-api-tls\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:10.748340 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:10.748309 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:11.041680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:11.041649 4409 patch_prober.go:29] interesting pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 07:28:11.042087 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:11.041706 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 07:28:11.161895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:11.161862 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr"] Nov 03 07:28:11.162212 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:28:11.162193 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533f3d9f_f1f3_47b1_8bfb_a145524ddd4f.slice/crio-ba9b37dfd702e937bd94e8b6aba0f9e875baa2a1afc373748d9a21ad3a114756 WatchSource:0}: Error finding container ba9b37dfd702e937bd94e8b6aba0f9e875baa2a1afc373748d9a21ad3a114756: Status 404 returned error can't find the container with id ba9b37dfd702e937bd94e8b6aba0f9e875baa2a1afc373748d9a21ad3a114756 Nov 03 07:28:12.031657 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:12.031624 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" event={"ID":"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f","Type":"ContainerStarted","Data":"ba9b37dfd702e937bd94e8b6aba0f9e875baa2a1afc373748d9a21ad3a114756"} Nov 03 07:28:12.041541 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:12.041516 4409 patch_prober.go:29] interesting pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 07:28:12.041658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:12.041578 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 07:28:13.050148 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:13.050103 4409 patch_prober.go:29] interesting pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 07:28:13.050912 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:13.050850 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 07:28:13.059667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:13.058333 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" event={"ID":"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f","Type":"ContainerStarted","Data":"bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178"} Nov 03 07:28:13.059667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:13.059611 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:13.247936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:13.247845 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" podStartSLOduration=3.4611175530000002 podStartE2EDuration="4.247824268s" totalImagesPullingTime="786.706715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:13.226286337 +0000 UTC m=+1748.418828089" watchObservedRunningTime="2026-11-03 07:28:13.247824268 +0000 UTC m=+1748.440366018" Nov 03 07:28:14.040977 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:14.040940 4409 patch_prober.go:29] interesting pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 07:28:14.041152 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:14.041000 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 07:28:15.041254 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:15.041221 4409 patch_prober.go:29] interesting pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 03 07:28:15.041667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:15.041276 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 03 07:28:16.045464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:16.045435 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:28:16.045917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:16.045728 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:28:16.046451 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:16.046431 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:28:20.102489 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:20.102419 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:28:24.514027 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:24.513994 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v"] Nov 03 07:28:24.514465 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:24.514235 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" podUID="89918fbd-c023-4f3a-9e09-26f620814a79" containerName="maas-api" containerID="cri-o://884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a" gracePeriod=30 Nov 03 07:28:25.117237 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.117183 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:28:25.204372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.204295 4409 generic.go:356] "Generic (PLEG): container finished" podID="89918fbd-c023-4f3a-9e09-26f620814a79" containerID="884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a" exitCode=0 Nov 03 07:28:25.204537 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.204522 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" event={"ID":"89918fbd-c023-4f3a-9e09-26f620814a79","Type":"ContainerDied","Data":"884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a"} Nov 03 07:28:25.204679 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.204627 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" event={"ID":"89918fbd-c023-4f3a-9e09-26f620814a79","Type":"ContainerDied","Data":"e98c23880a2dcbf8f1cc6550067965268bc2c4722a134e9b9fe99bb69de4a751"} Nov 03 07:28:25.204823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.204655 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v" Nov 03 07:28:25.205006 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.204689 4409 scope.go:122] "RemoveContainer" containerID="884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a" Nov 03 07:28:25.205227 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.204775 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89918fbd-c023-4f3a-9e09-26f620814a79-kube-api-access-zs2sv\" (UniqueName: \"kubernetes.io/projected/89918fbd-c023-4f3a-9e09-26f620814a79-kube-api-access-zs2sv\") pod \"89918fbd-c023-4f3a-9e09-26f620814a79\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " Nov 03 07:28:25.205655 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.205636 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-maas-api-tls\") pod \"89918fbd-c023-4f3a-9e09-26f620814a79\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " Nov 03 07:28:25.206228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.206195 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89918fbd-c023-4f3a-9e09-26f620814a79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89918fbd-c023-4f3a-9e09-26f620814a79-openshift-service-ca\") pod \"89918fbd-c023-4f3a-9e09-26f620814a79\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " Nov 03 07:28:25.206335 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.206306 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-metrics-certs\") pod \"89918fbd-c023-4f3a-9e09-26f620814a79\" (UID: \"89918fbd-c023-4f3a-9e09-26f620814a79\") " Nov 03 07:28:25.208156 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.208098 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89918fbd-c023-4f3a-9e09-26f620814a79-openshift-service-ca" pod "89918fbd-c023-4f3a-9e09-26f620814a79" (UID: "89918fbd-c023-4f3a-9e09-26f620814a79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:25.209272 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.209248 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89918fbd-c023-4f3a-9e09-26f620814a79-kube-api-access-zs2sv" pod "89918fbd-c023-4f3a-9e09-26f620814a79" (UID: "89918fbd-c023-4f3a-9e09-26f620814a79"). InnerVolumeSpecName "kube-api-access-zs2sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:25.209621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.209511 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-maas-api-tls" pod "89918fbd-c023-4f3a-9e09-26f620814a79" (UID: "89918fbd-c023-4f3a-9e09-26f620814a79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:28:25.209914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.209891 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-metrics-certs" pod "89918fbd-c023-4f3a-9e09-26f620814a79" (UID: "89918fbd-c023-4f3a-9e09-26f620814a79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:28:25.228189 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.228168 4409 scope.go:122] "RemoveContainer" containerID="884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a" Nov 03 07:28:25.228447 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:28:25.228428 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a\": container with ID starting with 884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a not found: ID does not exist" containerID="884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a" Nov 03 07:28:25.228507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.228454 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a"} err="failed to get container status \"884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a\": rpc error: code = NotFound desc = could not find container \"884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a\": container with ID starting with 884eb405b502cff85282ba9c174c0dc22cefd0e570083bec333265e096bc0e9a not found: ID does not exist" Nov 03 07:28:25.308914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.308888 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89918fbd-c023-4f3a-9e09-26f620814a79-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:25.308914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.308911 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:25.309093 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.308922 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs2sv\" (UniqueName: \"kubernetes.io/projected/89918fbd-c023-4f3a-9e09-26f620814a79-kube-api-access-zs2sv\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:25.309093 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.308931 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89918fbd-c023-4f3a-9e09-26f620814a79-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:25.797820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.797560 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v"] Nov 03 07:28:25.828785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:25.825802 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6d089af5-68c9bccd6d-d4p6v"] Nov 03 07:28:26.523682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:26.523647 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q"] Nov 03 07:28:26.523898 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:26.523877 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" containerID="cri-o://976e956f48846d56b062924f716f399ae726ff11611f026f0cf9c05772e2a4f6" gracePeriod=30 Nov 03 07:28:26.914608 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:26.914576 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89918fbd-c023-4f3a-9e09-26f620814a79" path="/var/lib/kubelet/pods/89918fbd-c023-4f3a-9e09-26f620814a79/volumes" Nov 03 07:28:27.461449 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:27.461400 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr"] Nov 03 07:28:27.461726 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:27.461673 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" containerID="cri-o://ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec" gracePeriod=30 Nov 03 07:28:28.298000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:28.297958 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:28:28.298497 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:28.298025 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:28:31.316430 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:31.316387 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2"] Nov 03 07:28:31.316964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:31.316639 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" podUID="12b0558f-1947-4064-aa94-d936b7f3206c" containerName="maas-api" containerID="cri-o://3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39" gracePeriod=30 Nov 03 07:28:32.147641 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.147580 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:28:32.447425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.388096 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b0558f-1947-4064-aa94-d936b7f3206c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b0558f-1947-4064-aa94-d936b7f3206c-openshift-service-ca\") pod \"12b0558f-1947-4064-aa94-d936b7f3206c\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " Nov 03 07:28:32.447425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.388312 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-maas-api-tls\") pod \"12b0558f-1947-4064-aa94-d936b7f3206c\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " Nov 03 07:28:32.447425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.388401 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b0558f-1947-4064-aa94-d936b7f3206c-kube-api-access-2l5cg\" (UniqueName: \"kubernetes.io/projected/12b0558f-1947-4064-aa94-d936b7f3206c-kube-api-access-2l5cg\") pod \"12b0558f-1947-4064-aa94-d936b7f3206c\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " Nov 03 07:28:32.447425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.388552 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-metrics-certs\") pod \"12b0558f-1947-4064-aa94-d936b7f3206c\" (UID: \"12b0558f-1947-4064-aa94-d936b7f3206c\") " Nov 03 07:28:32.447425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.414342 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b0558f-1947-4064-aa94-d936b7f3206c-openshift-service-ca" pod "12b0558f-1947-4064-aa94-d936b7f3206c" (UID: "12b0558f-1947-4064-aa94-d936b7f3206c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:32.447425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.433983 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-maas-api-tls" pod "12b0558f-1947-4064-aa94-d936b7f3206c" (UID: "12b0558f-1947-4064-aa94-d936b7f3206c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:28:32.520430 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.478605 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b0558f-1947-4064-aa94-d936b7f3206c-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.536760 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.528354 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.593588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.593545 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b0558f-1947-4064-aa94-d936b7f3206c-kube-api-access-2l5cg" pod "12b0558f-1947-4064-aa94-d936b7f3206c" (UID: "12b0558f-1947-4064-aa94-d936b7f3206c"). InnerVolumeSpecName "kube-api-access-2l5cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:32.631100 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.602973 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-metrics-certs" pod "12b0558f-1947-4064-aa94-d936b7f3206c" (UID: "12b0558f-1947-4064-aa94-d936b7f3206c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:28:32.700106 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.687868 4409 generic.go:356] "Generic (PLEG): container finished" podID="12b0558f-1947-4064-aa94-d936b7f3206c" containerID="3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39" exitCode=0 Nov 03 07:28:32.700106 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.688177 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" Nov 03 07:28:32.700106 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.697204 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" event={"ID":"12b0558f-1947-4064-aa94-d936b7f3206c","Type":"ContainerDied","Data":"3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39"} Nov 03 07:28:32.700106 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.697274 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2" event={"ID":"12b0558f-1947-4064-aa94-d936b7f3206c","Type":"ContainerDied","Data":"2f7dc3f6120d8d8ce8d463c90e807df3b5d635b6c52cfceb60642810c7f45235"} Nov 03 07:28:32.700106 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.697305 4409 scope.go:122] "RemoveContainer" containerID="3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39" Nov 03 07:28:32.703347 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.703317 4409 generic.go:356] "Generic (PLEG): container finished" podID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerID="976e956f48846d56b062924f716f399ae726ff11611f026f0cf9c05772e2a4f6" exitCode=0 Nov 03 07:28:32.703483 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.703381 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" event={"ID":"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf","Type":"ContainerDied","Data":"976e956f48846d56b062924f716f399ae726ff11611f026f0cf9c05772e2a4f6"} Nov 03 07:28:32.709839 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.709802 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:28:32.713889 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.713868 4409 scope.go:122] "RemoveContainer" containerID="3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39" Nov 03 07:28:32.714186 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:28:32.714161 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39\": container with ID starting with 3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39 not found: ID does not exist" containerID="3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39" Nov 03 07:28:32.714258 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.714196 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39"} err="failed to get container status \"3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39\": rpc error: code = NotFound desc = could not find container \"3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39\": container with ID starting with 3f7aa8db45844752fecfa6b417fe52075dee4a4556d6e3f749750886ceef1e39 not found: ID does not exist" Nov 03 07:28:32.732870 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.732841 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l5cg\" (UniqueName: \"kubernetes.io/projected/12b0558f-1947-4064-aa94-d936b7f3206c-kube-api-access-2l5cg\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.732972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.732876 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b0558f-1947-4064-aa94-d936b7f3206c-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.833844 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.833818 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istiod-ca-cert\") pod \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " Nov 03 07:28:32.834029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.833852 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-podinfo\") pod \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " Nov 03 07:28:32.834029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.833880 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-token\" (UniqueName: \"kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-token\") pod \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " Nov 03 07:28:32.834029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.833904 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-envoy\") pod \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " Nov 03 07:28:32.834029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.834012 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-certs\") pod \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " Nov 03 07:28:32.834268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.834108 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-kube-api-access-s2dc7\" (UniqueName: \"kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-kube-api-access-s2dc7\") pod \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " Nov 03 07:28:32.834268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.834212 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-data\") pod \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " Nov 03 07:28:32.834413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.834393 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-credential-socket\") pod \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " Nov 03 07:28:32.834591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.834401 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-certs" pod "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" (UID: "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.834682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.834427 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istiod-ca-cert" pod "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" (UID: "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:32.834682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.834611 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-credential-socket" pod "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" (UID: "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.834682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.834612 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-socket\") pod \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\" (UID: \"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf\") " Nov 03 07:28:32.834875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.834753 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-data" pod "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" (UID: "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.834875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.834826 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-socket" pod "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" (UID: "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.835313 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.835293 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.835419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.835315 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.835419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.835361 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.835419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.835378 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.835419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.835393 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.837326 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.837294 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-kube-api-access-s2dc7" pod "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" (UID: "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf"). InnerVolumeSpecName "kube-api-access-s2dc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:32.837432 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.837419 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-envoy" pod "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" (UID: "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.838009 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.837983 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-token" pod "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" (UID: "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:32.839201 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.839181 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-podinfo" pod "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" (UID: "23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:32.913887 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.913861 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:28:32.936081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936047 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-certs\") pod \"9d107dd0-ba31-4e3c-80fb-377d91700524\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " Nov 03 07:28:32.936215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936151 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-socket\") pod \"9d107dd0-ba31-4e3c-80fb-377d91700524\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " Nov 03 07:28:32.936215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936185 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-credential-socket\") pod \"9d107dd0-ba31-4e3c-80fb-377d91700524\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " Nov 03 07:28:32.936331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936232 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-data\") pod \"9d107dd0-ba31-4e3c-80fb-377d91700524\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " Nov 03 07:28:32.936331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936267 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-envoy\") pod \"9d107dd0-ba31-4e3c-80fb-377d91700524\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " Nov 03 07:28:32.936331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936298 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-token\" (UniqueName: \"kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-token\") pod \"9d107dd0-ba31-4e3c-80fb-377d91700524\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " Nov 03 07:28:32.936331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936298 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-certs" pod "9d107dd0-ba31-4e3c-80fb-377d91700524" (UID: "9d107dd0-ba31-4e3c-80fb-377d91700524"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.936526 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936413 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d107dd0-ba31-4e3c-80fb-377d91700524-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d107dd0-ba31-4e3c-80fb-377d91700524-istiod-ca-cert\") pod \"9d107dd0-ba31-4e3c-80fb-377d91700524\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " Nov 03 07:28:32.936526 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936419 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-socket" pod "9d107dd0-ba31-4e3c-80fb-377d91700524" (UID: "9d107dd0-ba31-4e3c-80fb-377d91700524"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.936526 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936503 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-kube-api-access-tzm98\" (UniqueName: \"kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-kube-api-access-tzm98\") pod \"9d107dd0-ba31-4e3c-80fb-377d91700524\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " Nov 03 07:28:32.936526 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936513 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-credential-socket" pod "9d107dd0-ba31-4e3c-80fb-377d91700524" (UID: "9d107dd0-ba31-4e3c-80fb-377d91700524"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.936749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936530 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-data" pod "9d107dd0-ba31-4e3c-80fb-377d91700524" (UID: "9d107dd0-ba31-4e3c-80fb-377d91700524"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:32.936749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936621 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-podinfo\") pod \"9d107dd0-ba31-4e3c-80fb-377d91700524\" (UID: \"9d107dd0-ba31-4e3c-80fb-377d91700524\") " Nov 03 07:28:32.936749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.936739 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d107dd0-ba31-4e3c-80fb-377d91700524-istiod-ca-cert" pod "9d107dd0-ba31-4e3c-80fb-377d91700524" (UID: "9d107dd0-ba31-4e3c-80fb-377d91700524"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:32.937359 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.937337 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2dc7\" (UniqueName: \"kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-kube-api-access-s2dc7\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.937359 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.937359 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.937515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.937368 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.937515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.937377 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.937515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.937385 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.937515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.937393 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.937515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.937401 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.937515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.937414 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d107dd0-ba31-4e3c-80fb-377d91700524-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.937515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.937427 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:32.938921 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.938897 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-kube-api-access-tzm98" pod "9d107dd0-ba31-4e3c-80fb-377d91700524" (UID: "9d107dd0-ba31-4e3c-80fb-377d91700524"). InnerVolumeSpecName "kube-api-access-tzm98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:32.939156 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.939116 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-podinfo" pod "9d107dd0-ba31-4e3c-80fb-377d91700524" (UID: "9d107dd0-ba31-4e3c-80fb-377d91700524"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:32.939275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.939254 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-token" pod "9d107dd0-ba31-4e3c-80fb-377d91700524" (UID: "9d107dd0-ba31-4e3c-80fb-377d91700524"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:32.939989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:32.939963 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-envoy" pod "9d107dd0-ba31-4e3c-80fb-377d91700524" (UID: "9d107dd0-ba31-4e3c-80fb-377d91700524"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:33.020566 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.020473 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2"] Nov 03 07:28:33.029395 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.029361 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-209c4c70-5dc6578b9b-49pm2"] Nov 03 07:28:33.039573 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.039543 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzm98\" (UniqueName: \"kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-kube-api-access-tzm98\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:33.039684 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.039577 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:33.039684 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.039593 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:33.039684 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.039602 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d107dd0-ba31-4e3c-80fb-377d91700524-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:33.712588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.712511 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" Nov 03 07:28:33.712588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.712531 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q" event={"ID":"23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf","Type":"ContainerDied","Data":"d7acc8245d6b77b7c751a2b00a85ce15d02909c3fbd6432991708e7779af4da1"} Nov 03 07:28:33.713007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.712600 4409 scope.go:122] "RemoveContainer" containerID="976e956f48846d56b062924f716f399ae726ff11611f026f0cf9c05772e2a4f6" Nov 03 07:28:33.716104 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.716074 4409 generic.go:356] "Generic (PLEG): container finished" podID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerID="ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec" exitCode=0 Nov 03 07:28:33.716259 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.716116 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" event={"ID":"9d107dd0-ba31-4e3c-80fb-377d91700524","Type":"ContainerDied","Data":"ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec"} Nov 03 07:28:33.716259 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.716127 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" Nov 03 07:28:33.716259 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.716158 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr" event={"ID":"9d107dd0-ba31-4e3c-80fb-377d91700524","Type":"ContainerDied","Data":"8b553cb9d3cee29fff836eb6b9159f664a9b50565c7901ba73596ee99ccc08e2"} Nov 03 07:28:33.724162 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.724147 4409 scope.go:122] "RemoveContainer" containerID="ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec" Nov 03 07:28:33.733852 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.733836 4409 scope.go:122] "RemoveContainer" containerID="77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f" Nov 03 07:28:33.745538 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.745520 4409 scope.go:122] "RemoveContainer" containerID="ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec" Nov 03 07:28:33.745840 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:28:33.745816 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec\": container with ID starting with ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec not found: ID does not exist" containerID="ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec" Nov 03 07:28:33.745934 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.745856 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec"} err="failed to get container status \"ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec\": rpc error: code = NotFound desc = could not find container \"ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec\": container with ID starting with ce284b30321094238a442a879ea1908c1d7fe6600be358243cb85e1a689ee5ec not found: ID does not exist" Nov 03 07:28:33.745934 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.745882 4409 scope.go:122] "RemoveContainer" containerID="77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f" Nov 03 07:28:33.746153 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:28:33.746135 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f\": container with ID starting with 77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f not found: ID does not exist" containerID="77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f" Nov 03 07:28:33.746194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.746159 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f"} err="failed to get container status \"77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f\": rpc error: code = NotFound desc = could not find container \"77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f\": container with ID starting with 77f613b8ed6a876061f51c77428b671cda98915fe31972be81561e92ad7d8a5f not found: ID does not exist" Nov 03 07:28:33.824658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.824626 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl"] Nov 03 07:28:33.824912 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.824889 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" containerID="cri-o://c5550701f39a8f14aa7d92fbbb631894233dd631c8fc3923b0cc40261a5d8256" gracePeriod=30 Nov 03 07:28:33.903744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.901802 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr"] Nov 03 07:28:33.944745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.942062 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6d089af5-openshift-default-66c5f9b66c-xgbwr"] Nov 03 07:28:33.956895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.956863 4409 patch_prober.go:29] interesting pod/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:28:33.957160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:33.957135 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:28:34.086805 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:34.086769 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q"] Nov 03 07:28:34.130756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:34.118119 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-rgz7q"] Nov 03 07:28:34.915617 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:34.915581 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b0558f-1947-4064-aa94-d936b7f3206c" path="/var/lib/kubelet/pods/12b0558f-1947-4064-aa94-d936b7f3206c/volumes" Nov 03 07:28:34.916160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:34.916141 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" path="/var/lib/kubelet/pods/23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf/volumes" Nov 03 07:28:34.916587 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:34.916570 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" path="/var/lib/kubelet/pods/9d107dd0-ba31-4e3c-80fb-377d91700524/volumes" Nov 03 07:28:35.382740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.381992 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g"] Nov 03 07:28:35.391095 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391058 4409 cpu_manager.go:414] "containerMap: removing container" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" Nov 03 07:28:35.391095 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391092 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" Nov 03 07:28:35.391371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391145 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" Nov 03 07:28:35.391371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391153 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" Nov 03 07:28:35.391371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391172 4409 cpu_manager.go:414] "containerMap: removing container" podUID="89918fbd-c023-4f3a-9e09-26f620814a79" containerName="maas-api" Nov 03 07:28:35.391371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391180 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89918fbd-c023-4f3a-9e09-26f620814a79" containerName="maas-api" Nov 03 07:28:35.391371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391192 4409 cpu_manager.go:414] "containerMap: removing container" podUID="12b0558f-1947-4064-aa94-d936b7f3206c" containerName="maas-api" Nov 03 07:28:35.391371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391200 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b0558f-1947-4064-aa94-d936b7f3206c" containerName="maas-api" Nov 03 07:28:35.391371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391236 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" Nov 03 07:28:35.391371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391243 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" Nov 03 07:28:35.391899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391423 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" Nov 03 07:28:35.391899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391437 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="89918fbd-c023-4f3a-9e09-26f620814a79" containerName="maas-api" Nov 03 07:28:35.391899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391445 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b0558f-1947-4064-aa94-d936b7f3206c" containerName="maas-api" Nov 03 07:28:35.391899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391458 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d107dd0-ba31-4e3c-80fb-377d91700524" containerName="istio-proxy" Nov 03 07:28:35.391899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.391483 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fe03fb-cbcf-4bd8-89d5-7dbd81d6bfdf" containerName="istio-proxy" Nov 03 07:28:35.433736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.395641 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.611687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.466649 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g"] Nov 03 07:28:35.611687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.494054 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-envoy\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.611687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.494124 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-token\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.611687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.494396 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-podinfo\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.611687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.494450 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-certs\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.611687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.494584 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-socket\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.611687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.494643 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74v2z\" (UniqueName: \"kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-kube-api-access-74v2z\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.611687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.494674 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc8eec3-9498-4244-afe5-456b9989e5f9-istiod-ca-cert\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.611687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.494736 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-credential-socket\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.611687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.494802 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-data\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.611687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.525208 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-acdd08-openshift-default-dockercfg-9sqpd\"" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.646855 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-podinfo\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.646920 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-certs\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.647038 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-socket\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.647092 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74v2z\" (UniqueName: \"kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-kube-api-access-74v2z\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.647129 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc8eec3-9498-4244-afe5-456b9989e5f9-istiod-ca-cert\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.647168 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-credential-socket\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.666199 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-data\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.666292 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-envoy\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.666329 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-token\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.667847 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-certs\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.668159 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-socket\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.669074 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc8eec3-9498-4244-afe5-456b9989e5f9-istiod-ca-cert\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.669312 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-credential-socket\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.682698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.669609 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-data\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.887739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.878149 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-envoy\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:35.979888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:35.934676 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-podinfo\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:36.088347 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:36.088278 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74v2z\" (UniqueName: \"kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-kube-api-access-74v2z\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:36.113802 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:36.113206 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-token\") pod \"e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:36.257612 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:36.257534 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:36.795937 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:36.793928 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g"] Nov 03 07:28:36.800902 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:28:36.798234 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc8eec3_9498_4244_afe5_456b9989e5f9.slice/crio-9239bb4d6b44865705091e88f455811b0d9499eec62919c318b721446870b94f WatchSource:0}: Error finding container 9239bb4d6b44865705091e88f455811b0d9499eec62919c318b721446870b94f: Status 404 returned error can't find the container with id 9239bb4d6b44865705091e88f455811b0d9499eec62919c318b721446870b94f Nov 03 07:28:36.804456 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:36.804414 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:28:36.804531 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:36.804516 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:28:36.804586 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:36.804566 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:28:37.050959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:37.050876 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" event={"ID":"5cc8eec3-9498-4244-afe5-456b9989e5f9","Type":"ContainerStarted","Data":"d78af7f55956bbafcc7891881249ab41125e112f5db8fc8fed77b969a750ebe1"} Nov 03 07:28:37.050959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:37.050939 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" event={"ID":"5cc8eec3-9498-4244-afe5-456b9989e5f9","Type":"ContainerStarted","Data":"9239bb4d6b44865705091e88f455811b0d9499eec62919c318b721446870b94f"} Nov 03 07:28:37.252038 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:37.251978 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" podStartSLOduration=2.251963129 podStartE2EDuration="2.251963129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:37.199363811 +0000 UTC m=+1772.391905557" watchObservedRunningTime="2026-11-03 07:28:37.251963129 +0000 UTC m=+1772.444504957" Nov 03 07:28:37.257906 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:37.257881 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:37.259211 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:37.259183 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 07:28:37.259308 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:37.259240 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 07:28:38.489740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:38.488260 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl"] Nov 03 07:28:38.668046 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:38.668012 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:38.668875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:38.668362 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" Nov 03 07:28:38.746218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:38.746127 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-5183ae\"" Nov 03 07:28:38.786958 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:38.748060 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-5183ae-dockercfg-tp7v4\"" Nov 03 07:28:38.841739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:38.822068 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl\" (UID: \"9027924d-f9a7-4a80-aff2-bd54b2cb04cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" Nov 03 07:28:38.841739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:38.822401 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzgpt\" (UniqueName: \"kubernetes.io/projected/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-kube-api-access-zzgpt\") pod \"payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl\" (UID: \"9027924d-f9a7-4a80-aff2-bd54b2cb04cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" Nov 03 07:28:38.863239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:38.861873 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl"] Nov 03 07:28:38.924604 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:38.923928 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzgpt\" (UniqueName: \"kubernetes.io/projected/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-kube-api-access-zzgpt\") pod \"payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl\" (UID: \"9027924d-f9a7-4a80-aff2-bd54b2cb04cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" Nov 03 07:28:38.924604 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:38.924060 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl\" (UID: \"9027924d-f9a7-4a80-aff2-bd54b2cb04cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" Nov 03 07:28:38.924944 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:38.924911 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl\" (UID: \"9027924d-f9a7-4a80-aff2-bd54b2cb04cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" Nov 03 07:28:39.001818 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:28:38.985908 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:28:39.019738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.018101 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzgpt\" (UniqueName: \"kubernetes.io/projected/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-kube-api-access-zzgpt\") pod \"payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl\" (UID: \"9027924d-f9a7-4a80-aff2-bd54b2cb04cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" Nov 03 07:28:39.134887 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.123119 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" Nov 03 07:28:39.196107 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.190155 4409 generic.go:356] "Generic (PLEG): container finished" podID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerID="c5550701f39a8f14aa7d92fbbb631894233dd631c8fc3923b0cc40261a5d8256" exitCode=0 Nov 03 07:28:39.196107 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.190314 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" event={"ID":"536748c7-12c7-4188-b0e4-0df4ee35a7ab","Type":"ContainerDied","Data":"c5550701f39a8f14aa7d92fbbb631894233dd631c8fc3923b0cc40261a5d8256"} Nov 03 07:28:39.196107 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.190456 4409 scope.go:122] "RemoveContainer" containerID="f2a640fc9d5663f7b395411c716c3ae5662a0ecd9ca93bd781fff9161639efab" Nov 03 07:28:39.196107 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.191174 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:39.210423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.209066 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:28:39.435506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.435142 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:28:39.452371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.452339 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn"] Nov 03 07:28:39.453148 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.453131 4409 cpu_manager.go:414] "containerMap: removing container" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" Nov 03 07:28:39.453148 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.453147 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" Nov 03 07:28:39.453319 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.453158 4409 cpu_manager.go:414] "containerMap: removing container" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" Nov 03 07:28:39.453319 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.453163 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" Nov 03 07:28:39.453319 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.453249 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" Nov 03 07:28:39.453319 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.453259 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" containerName="istio-proxy" Nov 03 07:28:39.457033 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.457004 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.513108 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.512586 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-09fd61-openshift-default-dockercfg-svknh\"" Nov 03 07:28:39.518897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.518852 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn"] Nov 03 07:28:39.563401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.563352 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-envoy\") pod \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " Nov 03 07:28:39.563598 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.563467 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istiod-ca-cert\") pod \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " Nov 03 07:28:39.564070 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.564025 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istiod-ca-cert" pod "536748c7-12c7-4188-b0e4-0df4ee35a7ab" (UID: "536748c7-12c7-4188-b0e4-0df4ee35a7ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:39.564223 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.564082 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-credential-socket\") pod \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " Nov 03 07:28:39.564223 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.564186 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-podinfo\") pod \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " Nov 03 07:28:39.564951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.564625 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-data\") pod \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " Nov 03 07:28:39.564951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.564767 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-certs\") pod \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " Nov 03 07:28:39.564951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.564881 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-token\" (UniqueName: \"kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-token\") pod \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " Nov 03 07:28:39.564951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.564916 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-kube-api-access-9grbv\" (UniqueName: \"kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-kube-api-access-9grbv\") pod \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " Nov 03 07:28:39.565261 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.564988 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-data" pod "536748c7-12c7-4188-b0e4-0df4ee35a7ab" (UID: "536748c7-12c7-4188-b0e4-0df4ee35a7ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:39.565261 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.565005 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-socket\") pod \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\" (UID: \"536748c7-12c7-4188-b0e4-0df4ee35a7ab\") " Nov 03 07:28:39.565689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.565416 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-credential-socket" pod "536748c7-12c7-4188-b0e4-0df4ee35a7ab" (UID: "536748c7-12c7-4188-b0e4-0df4ee35a7ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:39.565689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.565444 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istiod-ca-cert\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.565689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.565545 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-token\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.565895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.565728 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-socket\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.565895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.565774 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-credential-socket\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.565895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.565833 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-certs\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.566023 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.565936 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-envoy\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.566023 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.566006 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-podinfo\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.566106 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.566050 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-data\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.566152 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.566121 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz44k\" (UniqueName: \"kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-kube-api-access-bz44k\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.566349 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.566330 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:39.566499 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.566483 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:39.566616 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.566601 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:39.567018 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.566992 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-certs" pod "536748c7-12c7-4188-b0e4-0df4ee35a7ab" (UID: "536748c7-12c7-4188-b0e4-0df4ee35a7ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:39.567109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.567063 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-socket" pod "536748c7-12c7-4188-b0e4-0df4ee35a7ab" (UID: "536748c7-12c7-4188-b0e4-0df4ee35a7ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:39.576184 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.576156 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-podinfo" pod "536748c7-12c7-4188-b0e4-0df4ee35a7ab" (UID: "536748c7-12c7-4188-b0e4-0df4ee35a7ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:39.576294 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.576196 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-envoy" pod "536748c7-12c7-4188-b0e4-0df4ee35a7ab" (UID: "536748c7-12c7-4188-b0e4-0df4ee35a7ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:39.576961 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.576935 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-token" pod "536748c7-12c7-4188-b0e4-0df4ee35a7ab" (UID: "536748c7-12c7-4188-b0e4-0df4ee35a7ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670099 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-socket\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670165 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-credential-socket\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670205 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-certs\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670294 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-envoy\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670352 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-podinfo\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670388 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-data\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670449 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz44k\" (UniqueName: \"kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-kube-api-access-bz44k\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670547 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istiod-ca-cert\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670626 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-token\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670822 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670840 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670856 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670873 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536748c7-12c7-4188-b0e4-0df4ee35a7ab-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.670886 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536748c7-12c7-4188-b0e4-0df4ee35a7ab-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.671695 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-socket\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.672262 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-credential-socket\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.692750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.672557 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-certs\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.705799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.701319 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-data\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.705799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.703292 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istiod-ca-cert\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.705799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.704326 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-kube-api-access-9grbv" pod "536748c7-12c7-4188-b0e4-0df4ee35a7ab" (UID: "536748c7-12c7-4188-b0e4-0df4ee35a7ab"). InnerVolumeSpecName "kube-api-access-9grbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:39.772642 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.772611 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9grbv\" (UniqueName: \"kubernetes.io/projected/536748c7-12c7-4188-b0e4-0df4ee35a7ab-kube-api-access-9grbv\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:39.803451 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.803373 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-podinfo\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.804753 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.804670 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-token\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.809953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.809064 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl"] Nov 03 07:28:39.809953 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:28:39.809896 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9027924d_f9a7_4a80_aff2_bd54b2cb04cf.slice/crio-09d3ea01d049a11f3ad822248874379bff369ca65fb7f35bb4447c311c508614 WatchSource:0}: Error finding container 09d3ea01d049a11f3ad822248874379bff369ca65fb7f35bb4447c311c508614: Status 404 returned error can't find the container with id 09d3ea01d049a11f3ad822248874379bff369ca65fb7f35bb4447c311c508614 Nov 03 07:28:39.810546 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.810518 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz44k\" (UniqueName: \"kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-kube-api-access-bz44k\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:39.810650 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:39.810596 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-envoy\") pod \"e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:40.094941 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.094903 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:40.203918 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.203867 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" event={"ID":"9027924d-f9a7-4a80-aff2-bd54b2cb04cf","Type":"ContainerStarted","Data":"616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f"} Nov 03 07:28:40.204128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.203932 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" event={"ID":"9027924d-f9a7-4a80-aff2-bd54b2cb04cf","Type":"ContainerStarted","Data":"09d3ea01d049a11f3ad822248874379bff369ca65fb7f35bb4447c311c508614"} Nov 03 07:28:40.204128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.204099 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" Nov 03 07:28:40.206291 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.206219 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" event={"ID":"536748c7-12c7-4188-b0e4-0df4ee35a7ab","Type":"ContainerDied","Data":"71dc1efd6bbabe75d63520d2846e8daa91b971b0db58edcb50cdf365399af78e"} Nov 03 07:28:40.206443 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.206295 4409 scope.go:122] "RemoveContainer" containerID="c5550701f39a8f14aa7d92fbbb631894233dd631c8fc3923b0cc40261a5d8256" Nov 03 07:28:40.206443 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.206412 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl" Nov 03 07:28:40.328938 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.328877 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" podStartSLOduration=2.328859281 podStartE2EDuration="2.328859281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:40.322924028 +0000 UTC m=+1775.515465780" watchObservedRunningTime="2026-11-03 07:28:40.328859281 +0000 UTC m=+1775.521401034" Nov 03 07:28:40.463689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.463566 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl"] Nov 03 07:28:40.468196 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.468136 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-209c4c70-openshift-default-778f9c776f-9wnjl"] Nov 03 07:28:40.613902 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:28:40.613875 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7449c7e0_cb8b_47d0_9cd3_7bcd789b5a91.slice/crio-c4c144ac4e95d2fe8b85952a7a669eac26b328140a3c42f4b3cf56bca8603ebf WatchSource:0}: Error finding container c4c144ac4e95d2fe8b85952a7a669eac26b328140a3c42f4b3cf56bca8603ebf: Status 404 returned error can't find the container with id c4c144ac4e95d2fe8b85952a7a669eac26b328140a3c42f4b3cf56bca8603ebf Nov 03 07:28:40.615117 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.615088 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn"] Nov 03 07:28:40.616694 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.616660 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:28:40.616815 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.616773 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:28:40.616886 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.616815 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:28:40.917039 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:40.916992 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536748c7-12c7-4188-b0e4-0df4ee35a7ab" path="/var/lib/kubelet/pods/536748c7-12c7-4188-b0e4-0df4ee35a7ab/volumes" Nov 03 07:28:41.219325 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.219219 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" event={"ID":"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91","Type":"ContainerStarted","Data":"762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a"} Nov 03 07:28:41.219480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.219329 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" event={"ID":"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91","Type":"ContainerStarted","Data":"c4c144ac4e95d2fe8b85952a7a669eac26b328140a3c42f4b3cf56bca8603ebf"} Nov 03 07:28:41.348614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.348548 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" podStartSLOduration=2.348529693 podStartE2EDuration="2.348529693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:41.345169818 +0000 UTC m=+1776.537711567" watchObservedRunningTime="2026-11-03 07:28:41.348529693 +0000 UTC m=+1776.541071444" Nov 03 07:28:41.650768 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.647138 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4"] Nov 03 07:28:41.650768 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.647431 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" containerID="cri-o://419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a" gracePeriod=30 Nov 03 07:28:41.655835 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.653411 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77"] Nov 03 07:28:41.655835 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.653592 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerName="istio-proxy" containerID="cri-o://4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb" gracePeriod=30 Nov 03 07:28:41.660842 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.660810 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq"] Nov 03 07:28:41.662083 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.662051 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" containerID="cri-o://eb9f883d4804e759c4261fa38c4576097b598f43f1b4150abc0b997620564573" gracePeriod=30 Nov 03 07:28:41.664615 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.664588 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p"] Nov 03 07:28:41.664840 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.664819 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerName="istio-proxy" containerID="cri-o://ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f" gracePeriod=30 Nov 03 07:28:41.744857 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.744797 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg"] Nov 03 07:28:41.745588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.745472 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" containerID="cri-o://f51160c629c14e4a6f145bccfd9dd00ff61fb6013838cf55f38cfc4334124302" gracePeriod=30 Nov 03 07:28:41.750782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.750651 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9"] Nov 03 07:28:41.751096 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.751060 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" containerID="cri-o://bb2e13d35f6f0812aff7aee30f81bcc135989268a8ec294342cdbeed6c22c6ce" gracePeriod=30 Nov 03 07:28:41.756411 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.756033 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5"] Nov 03 07:28:41.756411 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.756255 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" containerID="cri-o://985dc45c9c9fa438a8e85db33f2778cd91dcb45047821b20559e6c576a597ba4" gracePeriod=30 Nov 03 07:28:41.760767 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.760725 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n"] Nov 03 07:28:41.760977 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:41.760954 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" podUID="459e890f-044a-4949-a9b8-6a1f40394cd0" containerName="istio-proxy" containerID="cri-o://cfb8c74287fc843902759272633fb6f3e5bc1817c2eaca7aa9fb0c9a00a683bf" gracePeriod=30 Nov 03 07:28:42.095974 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:42.095940 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:42.102627 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:42.102595 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:42.229951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:42.229911 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:42.231019 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:42.230999 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:28:42.290372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:42.290331 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z"] Nov 03 07:28:42.290599 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:42.290577 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" podUID="27d13c1e-7bc8-4579-9dd7-80c31d00cad2" containerName="istio-proxy" containerID="cri-o://f3d0ec3d40d1f3c5bdf9a26005627dc02c48692127afc44cb5c2c40671311e02" gracePeriod=30 Nov 03 07:28:42.336928 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:42.336887 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:28:42.337102 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:42.336958 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:28:44.098144 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:44.088240 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9"] Nov 03 07:28:44.114739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:44.101265 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" Nov 03 07:28:44.139741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:44.136472 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9"] Nov 03 07:28:44.160740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:44.159353 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-acdd08-dockercfg-dc94s\"" Nov 03 07:28:44.169738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:44.168410 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-acdd08\"" Nov 03 07:28:44.219745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:44.215006 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9\" (UID: \"51cec096-fc4b-4a5f-9b33-0c33c2a13b49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" Nov 03 07:28:44.219745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:44.215077 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjf6d\" (UniqueName: \"kubernetes.io/projected/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-kube-api-access-pjf6d\") pod \"payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9\" (UID: \"51cec096-fc4b-4a5f-9b33-0c33c2a13b49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" Nov 03 07:28:44.425799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:44.372309 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9\" (UID: \"51cec096-fc4b-4a5f-9b33-0c33c2a13b49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" Nov 03 07:28:44.441185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:44.426112 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjf6d\" (UniqueName: \"kubernetes.io/projected/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-kube-api-access-pjf6d\") pod \"payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9\" (UID: \"51cec096-fc4b-4a5f-9b33-0c33c2a13b49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" Nov 03 07:28:44.467177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:44.391961 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9\" (UID: \"51cec096-fc4b-4a5f-9b33-0c33c2a13b49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" Nov 03 07:28:44.630954 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:44.624526 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjf6d\" (UniqueName: \"kubernetes.io/projected/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-kube-api-access-pjf6d\") pod \"payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9\" (UID: \"51cec096-fc4b-4a5f-9b33-0c33c2a13b49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" Nov 03 07:28:44.856796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:44.813555 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" Nov 03 07:28:45.382484 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:45.382454 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9"] Nov 03 07:28:45.419735 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:28:45.419691 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cec096_fc4b_4a5f_9b33_0c33c2a13b49.slice/crio-b79560d8bcac9ba3286e731e5238c51ddd0c2b59e532079105fc291fee6db976 WatchSource:0}: Error finding container b79560d8bcac9ba3286e731e5238c51ddd0c2b59e532079105fc291fee6db976: Status 404 returned error can't find the container with id b79560d8bcac9ba3286e731e5238c51ddd0c2b59e532079105fc291fee6db976 Nov 03 07:28:45.470264 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:45.470229 4409 generic.go:356] "Generic (PLEG): container finished" podID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerID="b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd" exitCode=0 Nov 03 07:28:45.470467 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:45.470440 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" event={"ID":"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6","Type":"ContainerDied","Data":"b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd"} Nov 03 07:28:45.471004 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:45.470981 4409 scope.go:122] "RemoveContainer" containerID="b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd" Nov 03 07:28:45.472072 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:45.472047 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" event={"ID":"51cec096-fc4b-4a5f-9b33-0c33c2a13b49","Type":"ContainerStarted","Data":"b79560d8bcac9ba3286e731e5238c51ddd0c2b59e532079105fc291fee6db976"} Nov 03 07:28:45.472910 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:45.472880 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:28:45.473003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:45.472966 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:28:45.473065 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:45.473017 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:28:45.685480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:45.685387 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:28:45.685480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:45.685456 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" podUID="459e890f-044a-4949-a9b8-6a1f40394cd0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:28:46.295280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:46.294788 4409 patch_prober.go:29] interesting pod/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:28:46.295280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:46.294855 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:28:46.498406 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:46.498372 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" event={"ID":"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6","Type":"ContainerStarted","Data":"6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c"} Nov 03 07:28:46.500130 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:46.500103 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" event={"ID":"51cec096-fc4b-4a5f-9b33-0c33c2a13b49","Type":"ContainerStarted","Data":"eb0218ae7ca183e93ae8319fcf98f8e69ec4ca27873192e6446d61247690603a"} Nov 03 07:28:46.500256 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:46.500225 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" Nov 03 07:28:46.747138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:46.747067 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" podStartSLOduration=2.747049411 podStartE2EDuration="2.747049411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:46.742490087 +0000 UTC m=+1781.935031836" watchObservedRunningTime="2026-11-03 07:28:46.747049411 +0000 UTC m=+1781.939591161" Nov 03 07:28:47.013579 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.012895 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 07:28:47.023738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.012993 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 07:28:47.141331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.141295 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 07:28:47.141454 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.141356 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 07:28:47.211460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.211315 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:28:47.277325 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.277247 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:28:47.406562 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.406522 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-envoy\") pod \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " Nov 03 07:28:47.406842 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.406819 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-token\" (UniqueName: \"kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-token\") pod \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " Nov 03 07:28:47.407012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.406993 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-kube-api-access-j6xkr\" (UniqueName: \"kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-kube-api-access-j6xkr\") pod \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " Nov 03 07:28:47.415808 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.410799 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-podinfo\") pod \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " Nov 03 07:28:47.435913 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.435872 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-data\") pod \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " Nov 03 07:28:47.436172 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.436151 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-credential-socket\") pod \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " Nov 03 07:28:47.436267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.436193 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istiod-ca-cert\") pod \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " Nov 03 07:28:47.436267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.436244 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-certs\") pod \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " Nov 03 07:28:47.436372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.436309 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-socket\") pod \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\" (UID: \"ec563540-af7c-4fe7-8b7f-cc4be7102ab8\") " Nov 03 07:28:47.444077 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.444046 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-socket" pod "ec563540-af7c-4fe7-8b7f-cc4be7102ab8" (UID: "ec563540-af7c-4fe7-8b7f-cc4be7102ab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.444284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.444266 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-credential-socket" pod "ec563540-af7c-4fe7-8b7f-cc4be7102ab8" (UID: "ec563540-af7c-4fe7-8b7f-cc4be7102ab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.444441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.444425 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-data" pod "ec563540-af7c-4fe7-8b7f-cc4be7102ab8" (UID: "ec563540-af7c-4fe7-8b7f-cc4be7102ab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.444696 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.444679 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istiod-ca-cert" pod "ec563540-af7c-4fe7-8b7f-cc4be7102ab8" (UID: "ec563540-af7c-4fe7-8b7f-cc4be7102ab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:47.444968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.444943 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-certs" pod "ec563540-af7c-4fe7-8b7f-cc4be7102ab8" (UID: "ec563540-af7c-4fe7-8b7f-cc4be7102ab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.512966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.511349 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-kube-api-access-j6xkr" pod "ec563540-af7c-4fe7-8b7f-cc4be7102ab8" (UID: "ec563540-af7c-4fe7-8b7f-cc4be7102ab8"). InnerVolumeSpecName "kube-api-access-j6xkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.525665 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.517315 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-envoy" pod "ec563540-af7c-4fe7-8b7f-cc4be7102ab8" (UID: "ec563540-af7c-4fe7-8b7f-cc4be7102ab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.538755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.538704 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-podinfo" pod "ec563540-af7c-4fe7-8b7f-cc4be7102ab8" (UID: "ec563540-af7c-4fe7-8b7f-cc4be7102ab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:47.545120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.544204 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.545120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.544232 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.545120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.544246 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.545120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.544281 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6xkr\" (UniqueName: \"kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-kube-api-access-j6xkr\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.545120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.544295 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.545120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.544308 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.545120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.544321 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.545120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.544351 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.551159 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.551126 4409 generic.go:356] "Generic (PLEG): container finished" podID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerID="985dc45c9c9fa438a8e85db33f2778cd91dcb45047821b20559e6c576a597ba4" exitCode=0 Nov 03 07:28:47.551290 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.551232 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" event={"ID":"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8","Type":"ContainerDied","Data":"985dc45c9c9fa438a8e85db33f2778cd91dcb45047821b20559e6c576a597ba4"} Nov 03 07:28:47.553050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.552896 4409 generic.go:356] "Generic (PLEG): container finished" podID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerID="bb2e13d35f6f0812aff7aee30f81bcc135989268a8ec294342cdbeed6c22c6ce" exitCode=0 Nov 03 07:28:47.553050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.552993 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-token" pod "ec563540-af7c-4fe7-8b7f-cc4be7102ab8" (UID: "ec563540-af7c-4fe7-8b7f-cc4be7102ab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.553050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.553038 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" event={"ID":"702dd12a-8cf6-4daa-9530-8f85b6a8d01c","Type":"ContainerDied","Data":"bb2e13d35f6f0812aff7aee30f81bcc135989268a8ec294342cdbeed6c22c6ce"} Nov 03 07:28:47.553746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.553592 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:28:47.555761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.555159 4409 generic.go:356] "Generic (PLEG): container finished" podID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerID="f51160c629c14e4a6f145bccfd9dd00ff61fb6013838cf55f38cfc4334124302" exitCode=0 Nov 03 07:28:47.555761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.555282 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" event={"ID":"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4","Type":"ContainerDied","Data":"f51160c629c14e4a6f145bccfd9dd00ff61fb6013838cf55f38cfc4334124302"} Nov 03 07:28:47.556031 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.556005 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:28:47.557761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.556840 4409 generic.go:356] "Generic (PLEG): container finished" podID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerID="4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb" exitCode=0 Nov 03 07:28:47.557761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.556910 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" event={"ID":"ec563540-af7c-4fe7-8b7f-cc4be7102ab8","Type":"ContainerDied","Data":"4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb"} Nov 03 07:28:47.557761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.556945 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" event={"ID":"ec563540-af7c-4fe7-8b7f-cc4be7102ab8","Type":"ContainerDied","Data":"043a58359558a13463327b243809e3d49dbebb84388f48f365dfd2f1f5f30c47"} Nov 03 07:28:47.557761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.556973 4409 scope.go:122] "RemoveContainer" containerID="4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb" Nov 03 07:28:47.557761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.557120 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77" Nov 03 07:28:47.558103 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.558084 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:28:47.561553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.561130 4409 generic.go:356] "Generic (PLEG): container finished" podID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerID="419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a" exitCode=0 Nov 03 07:28:47.561553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.561200 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" event={"ID":"4f7ec919-d5c4-44e9-a498-82e604f21880","Type":"ContainerDied","Data":"419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a"} Nov 03 07:28:47.561553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.561239 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" event={"ID":"4f7ec919-d5c4-44e9-a498-82e604f21880","Type":"ContainerDied","Data":"a04ff7dcb6a395ade9caf6f838ea1ca929ebfc7cb2c30732b94affb429ce3755"} Nov 03 07:28:47.561553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.561307 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4" Nov 03 07:28:47.563858 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.563832 4409 generic.go:356] "Generic (PLEG): container finished" podID="27d13c1e-7bc8-4579-9dd7-80c31d00cad2" containerID="f3d0ec3d40d1f3c5bdf9a26005627dc02c48692127afc44cb5c2c40671311e02" exitCode=0 Nov 03 07:28:47.563972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.563954 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" event={"ID":"27d13c1e-7bc8-4579-9dd7-80c31d00cad2","Type":"ContainerDied","Data":"f3d0ec3d40d1f3c5bdf9a26005627dc02c48692127afc44cb5c2c40671311e02"} Nov 03 07:28:47.585735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.585483 4409 generic.go:356] "Generic (PLEG): container finished" podID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerID="eb9f883d4804e759c4261fa38c4576097b598f43f1b4150abc0b997620564573" exitCode=0 Nov 03 07:28:47.585735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.585583 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" event={"ID":"4eb42d5d-279b-4616-9b25-6d02bce7c1e3","Type":"ContainerDied","Data":"eb9f883d4804e759c4261fa38c4576097b598f43f1b4150abc0b997620564573"} Nov 03 07:28:47.589364 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.588971 4409 generic.go:356] "Generic (PLEG): container finished" podID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerID="ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f" exitCode=0 Nov 03 07:28:47.589364 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.589043 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" event={"ID":"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66","Type":"ContainerDied","Data":"ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f"} Nov 03 07:28:47.589364 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.589082 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" event={"ID":"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66","Type":"ContainerDied","Data":"c753b455bdeadf81b1d2582f161562bad3e709bfde9f029d49c9db31c637eeac"} Nov 03 07:28:47.589364 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.589218 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p" Nov 03 07:28:47.599582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.599560 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:28:47.601199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.601134 4409 generic.go:356] "Generic (PLEG): container finished" podID="459e890f-044a-4949-a9b8-6a1f40394cd0" containerID="cfb8c74287fc843902759272633fb6f3e5bc1817c2eaca7aa9fb0c9a00a683bf" exitCode=0 Nov 03 07:28:47.603338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.602516 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" event={"ID":"459e890f-044a-4949-a9b8-6a1f40394cd0","Type":"ContainerDied","Data":"cfb8c74287fc843902759272633fb6f3e5bc1817c2eaca7aa9fb0c9a00a683bf"} Nov 03 07:28:47.623702 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.623676 4409 scope.go:122] "RemoveContainer" containerID="4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb" Nov 03 07:28:47.624323 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:28:47.624296 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb\": container with ID starting with 4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb not found: ID does not exist" containerID="4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb" Nov 03 07:28:47.624390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.624339 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb"} err="failed to get container status \"4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb\": rpc error: code = NotFound desc = could not find container \"4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb\": container with ID starting with 4f0827e6f2639c04b0a476d1a4d519685d6735aef41dc05c92d8a3e14a7313cb not found: ID does not exist" Nov 03 07:28:47.624390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.624366 4409 scope.go:122] "RemoveContainer" containerID="419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a" Nov 03 07:28:47.641749 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.641318 4409 scope.go:122] "RemoveContainer" containerID="419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a" Nov 03 07:28:47.646506 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:28:47.646277 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a\": container with ID starting with 419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a not found: ID does not exist" containerID="419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a" Nov 03 07:28:47.646506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.646321 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a"} err="failed to get container status \"419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a\": rpc error: code = NotFound desc = could not find container \"419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a\": container with ID starting with 419652822ee7436f778d179074727ab38aaa6f194c7171236f3e5805d60e406a not found: ID does not exist" Nov 03 07:28:47.646506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.646345 4409 scope.go:122] "RemoveContainer" containerID="ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f" Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647050 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-token\" (UniqueName: \"kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-token\") pod \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647106 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-socket\") pod \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647139 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-data\") pod \"4f7ec919-d5c4-44e9-a498-82e604f21880\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647162 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-certs\") pod \"4f7ec919-d5c4-44e9-a498-82e604f21880\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647204 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-credential-socket\") pod \"4f7ec919-d5c4-44e9-a498-82e604f21880\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647308 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-credential-socket\") pod \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647340 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istiod-ca-cert\") pod \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647394 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istiod-ca-cert\") pod \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647421 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-kube-api-access-jbr69\" (UniqueName: \"kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-kube-api-access-jbr69\") pod \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647493 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-podinfo\") pod \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647545 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-kube-api-access-9dmwn\" (UniqueName: \"kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-kube-api-access-9dmwn\") pod \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647580 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-envoy\") pod \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647621 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7ec919-d5c4-44e9-a498-82e604f21880-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7ec919-d5c4-44e9-a498-82e604f21880-istiod-ca-cert\") pod \"4f7ec919-d5c4-44e9-a498-82e604f21880\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647667 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-token\" (UniqueName: \"kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-token\") pod \"4f7ec919-d5c4-44e9-a498-82e604f21880\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647697 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-envoy\") pod \"4f7ec919-d5c4-44e9-a498-82e604f21880\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647753 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-socket\") pod \"4f7ec919-d5c4-44e9-a498-82e604f21880\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.647865 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-credential-socket" pod "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" (UID: "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648041 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-podinfo\") pod \"4f7ec919-d5c4-44e9-a498-82e604f21880\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648050 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-socket" pod "4f7ec919-d5c4-44e9-a498-82e604f21880" (UID: "4f7ec919-d5c4-44e9-a498-82e604f21880"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648119 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-certs\") pod \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648155 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-socket\") pod \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648191 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-kube-api-access-s4db9\" (UniqueName: \"kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-kube-api-access-s4db9\") pod \"4f7ec919-d5c4-44e9-a498-82e604f21880\" (UID: \"4f7ec919-d5c4-44e9-a498-82e604f21880\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648264 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-certs\") pod \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648319 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istiod-ca-cert" pod "702dd12a-8cf6-4daa-9530-8f85b6a8d01c" (UID: "702dd12a-8cf6-4daa-9530-8f85b6a8d01c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648339 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-envoy\") pod \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648389 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-data\") pod \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648416 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-data\") pod \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648456 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-podinfo\") pod \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648500 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-token\" (UniqueName: \"kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-token\") pod \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\" (UID: \"1c7f6705-eb50-49b1-bb3c-dcd0761f2c66\") " Nov 03 07:28:47.648522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648539 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-credential-socket\") pod \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\" (UID: \"702dd12a-8cf6-4daa-9530-8f85b6a8d01c\") " Nov 03 07:28:47.652964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.648641 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istiod-ca-cert" pod "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" (UID: "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:47.652964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.649435 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec563540-af7c-4fe7-8b7f-cc4be7102ab8-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.652964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.649459 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.652964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.649475 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.652964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.649490 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.652964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.649503 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.661733 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.661296 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-envoy" pod "4f7ec919-d5c4-44e9-a498-82e604f21880" (UID: "4f7ec919-d5c4-44e9-a498-82e604f21880"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.661733 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.661505 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7ec919-d5c4-44e9-a498-82e604f21880-istiod-ca-cert" pod "4f7ec919-d5c4-44e9-a498-82e604f21880" (UID: "4f7ec919-d5c4-44e9-a498-82e604f21880"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:47.661893 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.661873 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-credential-socket" pod "702dd12a-8cf6-4daa-9530-8f85b6a8d01c" (UID: "702dd12a-8cf6-4daa-9530-8f85b6a8d01c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.662155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.662130 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-socket" pod "702dd12a-8cf6-4daa-9530-8f85b6a8d01c" (UID: "702dd12a-8cf6-4daa-9530-8f85b6a8d01c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.704890 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.682907 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-kube-api-access-9dmwn" pod "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" (UID: "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66"). InnerVolumeSpecName "kube-api-access-9dmwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.714889 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.714856 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-data" pod "702dd12a-8cf6-4daa-9530-8f85b6a8d01c" (UID: "702dd12a-8cf6-4daa-9530-8f85b6a8d01c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.715242 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.715222 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-certs" pod "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" (UID: "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.717544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.717359 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-data" pod "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" (UID: "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.717544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.717483 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-socket" pod "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" (UID: "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.720333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.718280 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-certs" pod "702dd12a-8cf6-4daa-9530-8f85b6a8d01c" (UID: "702dd12a-8cf6-4daa-9530-8f85b6a8d01c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.720333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.718992 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-credential-socket" pod "4f7ec919-d5c4-44e9-a498-82e604f21880" (UID: "4f7ec919-d5c4-44e9-a498-82e604f21880"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.720333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.719665 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-certs" pod "4f7ec919-d5c4-44e9-a498-82e604f21880" (UID: "4f7ec919-d5c4-44e9-a498-82e604f21880"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.720333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.719919 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-data" pod "4f7ec919-d5c4-44e9-a498-82e604f21880" (UID: "4f7ec919-d5c4-44e9-a498-82e604f21880"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.723687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.723557 4409 scope.go:122] "RemoveContainer" containerID="094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725" Nov 03 07:28:47.725920 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.725886 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-podinfo" pod "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" (UID: "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:47.726121 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.725953 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-podinfo" pod "4f7ec919-d5c4-44e9-a498-82e604f21880" (UID: "4f7ec919-d5c4-44e9-a498-82e604f21880"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:47.726387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.726368 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-envoy" pod "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" (UID: "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.726813 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.726759 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-token" pod "702dd12a-8cf6-4daa-9530-8f85b6a8d01c" (UID: "702dd12a-8cf6-4daa-9530-8f85b6a8d01c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.727417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.727362 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-podinfo" pod "702dd12a-8cf6-4daa-9530-8f85b6a8d01c" (UID: "702dd12a-8cf6-4daa-9530-8f85b6a8d01c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:47.727417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.727381 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-token" pod "4f7ec919-d5c4-44e9-a498-82e604f21880" (UID: "4f7ec919-d5c4-44e9-a498-82e604f21880"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.727557 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.727483 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-envoy" pod "702dd12a-8cf6-4daa-9530-8f85b6a8d01c" (UID: "702dd12a-8cf6-4daa-9530-8f85b6a8d01c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.728058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.728022 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-kube-api-access-jbr69" pod "702dd12a-8cf6-4daa-9530-8f85b6a8d01c" (UID: "702dd12a-8cf6-4daa-9530-8f85b6a8d01c"). InnerVolumeSpecName "kube-api-access-jbr69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.728668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.728618 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-token" pod "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" (UID: "1c7f6705-eb50-49b1-bb3c-dcd0761f2c66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.752553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.750581 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-kube-api-access-s4db9" pod "4f7ec919-d5c4-44e9-a498-82e604f21880" (UID: "4f7ec919-d5c4-44e9-a498-82e604f21880"). InnerVolumeSpecName "kube-api-access-s4db9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.752553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.751177 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-kube-api-access-r4wjt\" (UniqueName: \"kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-kube-api-access-r4wjt\") pod \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " Nov 03 07:28:47.752553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.751216 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-certs\") pod \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " Nov 03 07:28:47.752553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.751284 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-credential-socket\") pod \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " Nov 03 07:28:47.752553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.751383 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-podinfo\") pod \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " Nov 03 07:28:47.752553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.751472 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-envoy\") pod \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " Nov 03 07:28:47.752553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.751520 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-socket\") pod \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " Nov 03 07:28:47.752553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.751561 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-token\" (UniqueName: \"kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-token\") pod \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " Nov 03 07:28:47.752553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.752144 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-certs" pod "4eb42d5d-279b-4616-9b25-6d02bce7c1e3" (UID: "4eb42d5d-279b-4616-9b25-6d02bce7c1e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.754017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.753545 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-data" pod "4eb42d5d-279b-4616-9b25-6d02bce7c1e3" (UID: "4eb42d5d-279b-4616-9b25-6d02bce7c1e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.754017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.753908 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-socket" pod "4eb42d5d-279b-4616-9b25-6d02bce7c1e3" (UID: "4eb42d5d-279b-4616-9b25-6d02bce7c1e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.754945 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.751594 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-data\") pod \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " Nov 03 07:28:47.754945 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.754420 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istiod-ca-cert\") pod \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\" (UID: \"4eb42d5d-279b-4616-9b25-6d02bce7c1e3\") " Nov 03 07:28:47.755406 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.755383 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istiod-ca-cert" pod "4eb42d5d-279b-4616-9b25-6d02bce7c1e3" (UID: "4eb42d5d-279b-4616-9b25-6d02bce7c1e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756079 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756102 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756116 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756129 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756142 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756154 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756165 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756177 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756190 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756206 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756220 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756233 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756246 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756260 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbr69\" (UniqueName: \"kubernetes.io/projected/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-kube-api-access-jbr69\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756274 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756286 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dmwn\" (UniqueName: \"kubernetes.io/projected/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66-kube-api-access-9dmwn\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756298 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756312 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f7ec919-d5c4-44e9-a498-82e604f21880-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756326 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756340 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756353 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756370 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756384 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756399 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f7ec919-d5c4-44e9-a498-82e604f21880-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756412 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756425 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/702dd12a-8cf6-4daa-9530-8f85b6a8d01c-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.757630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.756438 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4db9\" (UniqueName: \"kubernetes.io/projected/4f7ec919-d5c4-44e9-a498-82e604f21880-kube-api-access-s4db9\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.759736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.759690 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-kube-api-access-r4wjt" pod "4eb42d5d-279b-4616-9b25-6d02bce7c1e3" (UID: "4eb42d5d-279b-4616-9b25-6d02bce7c1e3"). InnerVolumeSpecName "kube-api-access-r4wjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:47.784002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.783926 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-credential-socket" pod "4eb42d5d-279b-4616-9b25-6d02bce7c1e3" (UID: "4eb42d5d-279b-4616-9b25-6d02bce7c1e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:47.871736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.869349 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4wjt\" (UniqueName: \"kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-kube-api-access-r4wjt\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.871736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.869440 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:47.879954 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.879462 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 07:28:47.879954 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:47.879526 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 07:28:48.005658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.005614 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-envoy" pod "4eb42d5d-279b-4616-9b25-6d02bce7c1e3" (UID: "4eb42d5d-279b-4616-9b25-6d02bce7c1e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.014686 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.014651 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-podinfo" pod "4eb42d5d-279b-4616-9b25-6d02bce7c1e3" (UID: "4eb42d5d-279b-4616-9b25-6d02bce7c1e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:48.019576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.019557 4409 scope.go:122] "RemoveContainer" containerID="ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f" Nov 03 07:28:48.020624 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:28:48.020465 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f\": container with ID starting with ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f not found: ID does not exist" containerID="ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f" Nov 03 07:28:48.020624 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.020503 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f"} err="failed to get container status \"ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f\": rpc error: code = NotFound desc = could not find container \"ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f\": container with ID starting with ce8e72b263f3fc2fea3118635de6d82829de3fdfeb44b980f3d2800692864d8f not found: ID does not exist" Nov 03 07:28:48.020624 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.020525 4409 scope.go:122] "RemoveContainer" containerID="094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725" Nov 03 07:28:48.021160 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:28:48.021121 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725\": container with ID starting with 094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725 not found: ID does not exist" containerID="094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725" Nov 03 07:28:48.021259 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.021158 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725"} err="failed to get container status \"094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725\": rpc error: code = NotFound desc = could not find container \"094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725\": container with ID starting with 094d8d71f3691b2d689a241a5e5b9b3c219a23cbbd85c9cd1a211b831f8db725 not found: ID does not exist" Nov 03 07:28:48.022880 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.022856 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-token" pod "4eb42d5d-279b-4616-9b25-6d02bce7c1e3" (UID: "4eb42d5d-279b-4616-9b25-6d02bce7c1e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:48.040609 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.040590 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:28:48.050681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.050496 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:28:48.054581 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.054516 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:28:48.061587 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.061533 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:28:48.081979 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.080249 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.081979 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.080284 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.081979 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.080303 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb42d5d-279b-4616-9b25-6d02bce7c1e3-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183167 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-kube-api-access-jltx8\" (UniqueName: \"kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-kube-api-access-jltx8\") pod \"459e890f-044a-4949-a9b8-6a1f40394cd0\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183285 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-socket\") pod \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183322 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-certs\") pod \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183355 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-data\") pod \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183389 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-certs\") pod \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183433 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-podinfo\") pod \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183461 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-credential-socket\") pod \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183491 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-podinfo\") pod \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183543 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-certs\") pod \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183588 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-kube-api-access-6xsr2\" (UniqueName: \"kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-kube-api-access-6xsr2\") pod \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183640 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-kube-api-access-bkcgk\" (UniqueName: \"kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-kube-api-access-bkcgk\") pod \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183670 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-credential-socket\") pod \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183729 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-envoy\") pod \"459e890f-044a-4949-a9b8-6a1f40394cd0\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183779 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459e890f-044a-4949-a9b8-6a1f40394cd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459e890f-044a-4949-a9b8-6a1f40394cd0-istiod-ca-cert\") pod \"459e890f-044a-4949-a9b8-6a1f40394cd0\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183865 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-certs\") pod \"459e890f-044a-4949-a9b8-6a1f40394cd0\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183931 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-envoy\") pod \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.183987 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-credential-socket\") pod \"459e890f-044a-4949-a9b8-6a1f40394cd0\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.184028 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istiod-ca-cert\") pod \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.184081 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-credential-socket\") pod \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.184113 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-token\" (UniqueName: \"kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-token\") pod \"459e890f-044a-4949-a9b8-6a1f40394cd0\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.184172 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istiod-ca-cert\") pod \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.184225 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-socket\") pod \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " Nov 03 07:28:48.185736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.184264 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-data\") pod \"459e890f-044a-4949-a9b8-6a1f40394cd0\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " Nov 03 07:28:48.191932 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.191896 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-data\") pod \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " Nov 03 07:28:48.192150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.192116 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-envoy\") pod \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " Nov 03 07:28:48.192685 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.192329 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-podinfo\") pod \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.197359 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-token\" (UniqueName: \"kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-token\") pod \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.197447 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istiod-ca-cert\") pod \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.197490 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-podinfo\") pod \"459e890f-044a-4949-a9b8-6a1f40394cd0\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.197519 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-token\" (UniqueName: \"kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-token\") pod \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.197598 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-data\") pod \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\" (UID: \"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4\") " Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.197645 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-token\" (UniqueName: \"kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-token\") pod \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.197675 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-envoy\") pod \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.197702 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-kube-api-access-g7867\" (UniqueName: \"kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-kube-api-access-g7867\") pod \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\" (UID: \"27d13c1e-7bc8-4579-9dd7-80c31d00cad2\") " Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.197793 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-socket\") pod \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\" (UID: \"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8\") " Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.197883 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-socket\") pod \"459e890f-044a-4949-a9b8-6a1f40394cd0\" (UID: \"459e890f-044a-4949-a9b8-6a1f40394cd0\") " Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.198834 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-socket" pod "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" (UID: "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.199099 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-certs" pod "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" (UID: "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.199298 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-data" pod "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" (UID: "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.199761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.199476 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-certs" pod "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" (UID: "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.202768 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.201768 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-credential-socket" pod "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" (UID: "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.202768 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.202670 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-certs" pod "27d13c1e-7bc8-4579-9dd7-80c31d00cad2" (UID: "27d13c1e-7bc8-4579-9dd7-80c31d00cad2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.204276 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.204249 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-podinfo" pod "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" (UID: "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:48.205742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.205687 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-credential-socket" pod "27d13c1e-7bc8-4579-9dd7-80c31d00cad2" (UID: "27d13c1e-7bc8-4579-9dd7-80c31d00cad2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.205871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.205818 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-podinfo" pod "27d13c1e-7bc8-4579-9dd7-80c31d00cad2" (UID: "27d13c1e-7bc8-4579-9dd7-80c31d00cad2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:48.205949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.205861 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istiod-ca-cert" pod "27d13c1e-7bc8-4579-9dd7-80c31d00cad2" (UID: "27d13c1e-7bc8-4579-9dd7-80c31d00cad2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:48.206537 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.206498 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-socket" pod "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" (UID: "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.207619 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.207446 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-token" pod "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" (UID: "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:48.207619 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.207541 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-token" pod "27d13c1e-7bc8-4579-9dd7-80c31d00cad2" (UID: "27d13c1e-7bc8-4579-9dd7-80c31d00cad2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:48.207869 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.207659 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-socket" pod "459e890f-044a-4949-a9b8-6a1f40394cd0" (UID: "459e890f-044a-4949-a9b8-6a1f40394cd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.207869 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.207769 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-podinfo" pod "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" (UID: "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:48.207990 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.207961 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-data" pod "459e890f-044a-4949-a9b8-6a1f40394cd0" (UID: "459e890f-044a-4949-a9b8-6a1f40394cd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.208558 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.208282 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istiod-ca-cert" pod "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" (UID: "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:48.208558 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.208390 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istiod-ca-cert" pod "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" (UID: "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:48.208780 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.208560 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-kube-api-access-bkcgk" pod "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" (UID: "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8"). InnerVolumeSpecName "kube-api-access-bkcgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:48.208780 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.208587 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-socket" pod "27d13c1e-7bc8-4579-9dd7-80c31d00cad2" (UID: "27d13c1e-7bc8-4579-9dd7-80c31d00cad2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.208925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.208788 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-credential-socket" pod "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" (UID: "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.208925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.208795 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-data" pod "27d13c1e-7bc8-4579-9dd7-80c31d00cad2" (UID: "27d13c1e-7bc8-4579-9dd7-80c31d00cad2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.209123 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.209103 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459e890f-044a-4949-a9b8-6a1f40394cd0-istiod-ca-cert" pod "459e890f-044a-4949-a9b8-6a1f40394cd0" (UID: "459e890f-044a-4949-a9b8-6a1f40394cd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:28:48.209436 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.209251 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-certs" pod "459e890f-044a-4949-a9b8-6a1f40394cd0" (UID: "459e890f-044a-4949-a9b8-6a1f40394cd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.209612 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.209583 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-data" pod "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" (UID: "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.209762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.209660 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-credential-socket" pod "459e890f-044a-4949-a9b8-6a1f40394cd0" (UID: "459e890f-044a-4949-a9b8-6a1f40394cd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.211207 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.211173 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-token" pod "459e890f-044a-4949-a9b8-6a1f40394cd0" (UID: "459e890f-044a-4949-a9b8-6a1f40394cd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:48.211305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.211214 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-envoy" pod "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" (UID: "1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.212508 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.212481 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-kube-api-access-jltx8" pod "459e890f-044a-4949-a9b8-6a1f40394cd0" (UID: "459e890f-044a-4949-a9b8-6a1f40394cd0"). InnerVolumeSpecName "kube-api-access-jltx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:48.212616 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.212594 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-kube-api-access-6xsr2" pod "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" (UID: "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4"). InnerVolumeSpecName "kube-api-access-6xsr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:48.212699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.212683 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-envoy" pod "27d13c1e-7bc8-4579-9dd7-80c31d00cad2" (UID: "27d13c1e-7bc8-4579-9dd7-80c31d00cad2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.213054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.213034 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-envoy" pod "459e890f-044a-4949-a9b8-6a1f40394cd0" (UID: "459e890f-044a-4949-a9b8-6a1f40394cd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.213310 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.213271 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-podinfo" pod "459e890f-044a-4949-a9b8-6a1f40394cd0" (UID: "459e890f-044a-4949-a9b8-6a1f40394cd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:28:48.213403 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.213329 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-kube-api-access-g7867" pod "27d13c1e-7bc8-4579-9dd7-80c31d00cad2" (UID: "27d13c1e-7bc8-4579-9dd7-80c31d00cad2"). InnerVolumeSpecName "kube-api-access-g7867". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:48.213668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.213641 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-token" pod "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" (UID: "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:28:48.214862 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.214838 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-envoy" pod "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" (UID: "9d0484f1-4f69-4550-a16f-1ccc33b0c0f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:28:48.290534 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.290409 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77"] Nov 03 07:28:48.296206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.296176 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2ll77"] Nov 03 07:28:48.300307 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300282 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459e890f-044a-4949-a9b8-6a1f40394cd0-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300310 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300327 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300340 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300353 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300367 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300380 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300393 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300408 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300420 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300444 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300458 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300472 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300487 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300501 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300514 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300528 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300540 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300554 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300567 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300580 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7867\" (UniqueName: \"kubernetes.io/projected/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-kube-api-access-g7867\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300596 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300611 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300627 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jltx8\" (UniqueName: \"kubernetes.io/projected/459e890f-044a-4949-a9b8-6a1f40394cd0-kube-api-access-jltx8\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300642 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300656 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300670 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300683 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300697 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300754 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300769 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300781 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d13c1e-7bc8-4579-9dd7-80c31d00cad2-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300796 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xsr2\" (UniqueName: \"kubernetes.io/projected/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4-kube-api-access-6xsr2\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300811 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkcgk\" (UniqueName: \"kubernetes.io/projected/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-kube-api-access-bkcgk\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300824 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.300903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.300838 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459e890f-044a-4949-a9b8-6a1f40394cd0-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:28:48.455666 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.455635 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p"] Nov 03 07:28:48.461072 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.461044 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-s446p"] Nov 03 07:28:48.614294 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.614262 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" Nov 03 07:28:48.614294 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.614281 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg" event={"ID":"9d0484f1-4f69-4550-a16f-1ccc33b0c0f4","Type":"ContainerDied","Data":"0c4be6fe04b5d5f9533cba7ca9b0c77d5b73cc5e27aaf8d89d9c2c837b8ec39d"} Nov 03 07:28:48.614875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.614347 4409 scope.go:122] "RemoveContainer" containerID="f51160c629c14e4a6f145bccfd9dd00ff61fb6013838cf55f38cfc4334124302" Nov 03 07:28:48.623044 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.619475 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" Nov 03 07:28:48.623044 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.619494 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z" event={"ID":"27d13c1e-7bc8-4579-9dd7-80c31d00cad2","Type":"ContainerDied","Data":"8f45f1b5d68fbd292ff0acd53a726f00bc63c4c7f28cf30cd65c182f77b4209c"} Nov 03 07:28:48.623044 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.622863 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" event={"ID":"4eb42d5d-279b-4616-9b25-6d02bce7c1e3","Type":"ContainerDied","Data":"a71dd0cc06090aeab48aa7f530277ce8951e59791c8faa7a7ac36d9f5e3af0ad"} Nov 03 07:28:48.623044 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.622900 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq" Nov 03 07:28:48.632246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.632222 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" Nov 03 07:28:48.632246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.632241 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n" event={"ID":"459e890f-044a-4949-a9b8-6a1f40394cd0","Type":"ContainerDied","Data":"6640bca635f208622f11796f8cc57df9f276ab6236030621565ac51ac3c8a735"} Nov 03 07:28:48.633872 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.633697 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" event={"ID":"1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8","Type":"ContainerDied","Data":"df4a4f39c14b39f484410a4168147ca6766332a299cfa1dee5de30ad7d164ee7"} Nov 03 07:28:48.634793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.634454 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5" Nov 03 07:28:48.635157 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.635138 4409 scope.go:122] "RemoveContainer" containerID="f3d0ec3d40d1f3c5bdf9a26005627dc02c48692127afc44cb5c2c40671311e02" Nov 03 07:28:48.635985 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.635917 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" event={"ID":"702dd12a-8cf6-4daa-9530-8f85b6a8d01c","Type":"ContainerDied","Data":"9e474247c348cfed286958a24dbcf335ad290a863771a988bbc258f3567e91a0"} Nov 03 07:28:48.636075 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.635984 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9" Nov 03 07:28:48.646383 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.646334 4409 scope.go:122] "RemoveContainer" containerID="eb9f883d4804e759c4261fa38c4576097b598f43f1b4150abc0b997620564573" Nov 03 07:28:48.661820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.661781 4409 scope.go:122] "RemoveContainer" containerID="cfb8c74287fc843902759272633fb6f3e5bc1817c2eaca7aa9fb0c9a00a683bf" Nov 03 07:28:48.716216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.716190 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4"] Nov 03 07:28:48.728320 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.728294 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-2lpv4"] Nov 03 07:28:48.730356 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.730259 4409 scope.go:122] "RemoveContainer" containerID="985dc45c9c9fa438a8e85db33f2778cd91dcb45047821b20559e6c576a597ba4" Nov 03 07:28:48.743031 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.743010 4409 scope.go:122] "RemoveContainer" containerID="bb2e13d35f6f0812aff7aee30f81bcc135989268a8ec294342cdbeed6c22c6ce" Nov 03 07:28:48.892653 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.892553 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 07:28:48.892653 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.892621 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 07:28:48.916080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.916054 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" path="/var/lib/kubelet/pods/1c7f6705-eb50-49b1-bb3c-dcd0761f2c66/volumes" Nov 03 07:28:48.916753 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.916688 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" path="/var/lib/kubelet/pods/4f7ec919-d5c4-44e9-a498-82e604f21880/volumes" Nov 03 07:28:48.917440 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:48.917415 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" path="/var/lib/kubelet/pods/ec563540-af7c-4fe7-8b7f-cc4be7102ab8/volumes" Nov 03 07:28:49.355544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:49.355501 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n"] Nov 03 07:28:49.393277 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:49.393239 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-srf4n"] Nov 03 07:28:49.495660 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:49.495630 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq"] Nov 03 07:28:49.499729 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:49.499693 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-669d6b-openshift-default-548f97d447-xgkwq"] Nov 03 07:28:49.622461 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:49.622376 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z"] Nov 03 07:28:49.626465 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:49.626443 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-bkk6z"] Nov 03 07:28:49.753760 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:49.751974 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5"] Nov 03 07:28:49.792049 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:49.791977 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-8k9v5"] Nov 03 07:28:49.890247 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:49.890160 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 07:28:49.890522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:49.890488 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 07:28:49.892342 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:49.892315 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg"] Nov 03 07:28:49.896291 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:49.896269 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-2rlqg"] Nov 03 07:28:50.014030 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:50.014001 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9"] Nov 03 07:28:50.020350 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:50.020326 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-6q4r9"] Nov 03 07:28:50.884739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:50.881047 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 07:28:50.884739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:50.881103 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 07:28:50.914201 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:50.914169 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" path="/var/lib/kubelet/pods/1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8/volumes" Nov 03 07:28:50.914699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:50.914685 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d13c1e-7bc8-4579-9dd7-80c31d00cad2" path="/var/lib/kubelet/pods/27d13c1e-7bc8-4579-9dd7-80c31d00cad2/volumes" Nov 03 07:28:50.915178 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:50.915164 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459e890f-044a-4949-a9b8-6a1f40394cd0" path="/var/lib/kubelet/pods/459e890f-044a-4949-a9b8-6a1f40394cd0/volumes" Nov 03 07:28:50.915916 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:50.915901 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" path="/var/lib/kubelet/pods/4eb42d5d-279b-4616-9b25-6d02bce7c1e3/volumes" Nov 03 07:28:50.916321 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:50.916309 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" path="/var/lib/kubelet/pods/702dd12a-8cf6-4daa-9530-8f85b6a8d01c/volumes" Nov 03 07:28:50.916724 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:50.916702 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" path="/var/lib/kubelet/pods/9d0484f1-4f69-4550-a16f-1ccc33b0c0f4/volumes" Nov 03 07:28:51.224537 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.224460 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" Nov 03 07:28:51.768411 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.763941 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf"] Nov 03 07:28:51.769826 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.769732 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" Nov 03 07:28:51.770069 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.770048 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" Nov 03 07:28:51.770200 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.770188 4409 cpu_manager.go:414] "containerMap: removing container" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerName="istio-proxy" Nov 03 07:28:51.770284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.770274 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerName="istio-proxy" Nov 03 07:28:51.770371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.770361 4409 cpu_manager.go:414] "containerMap: removing container" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" Nov 03 07:28:51.770443 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.770432 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" Nov 03 07:28:51.770516 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.770507 4409 cpu_manager.go:414] "containerMap: removing container" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerName="istio-proxy" Nov 03 07:28:51.770590 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.770580 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerName="istio-proxy" Nov 03 07:28:51.770670 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.770661 4409 cpu_manager.go:414] "containerMap: removing container" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" Nov 03 07:28:51.770760 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.770747 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" Nov 03 07:28:51.770851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.770842 4409 cpu_manager.go:414] "containerMap: removing container" podUID="27d13c1e-7bc8-4579-9dd7-80c31d00cad2" containerName="istio-proxy" Nov 03 07:28:51.774767 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.774742 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d13c1e-7bc8-4579-9dd7-80c31d00cad2" containerName="istio-proxy" Nov 03 07:28:51.774925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.774910 4409 cpu_manager.go:414] "containerMap: removing container" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" Nov 03 07:28:51.775025 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.775009 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" Nov 03 07:28:51.775135 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.775124 4409 cpu_manager.go:414] "containerMap: removing container" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerName="istio-proxy" Nov 03 07:28:51.775210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.775200 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerName="istio-proxy" Nov 03 07:28:51.781315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.775298 4409 cpu_manager.go:414] "containerMap: removing container" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" Nov 03 07:28:51.781537 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.781477 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" Nov 03 07:28:51.781657 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.781645 4409 cpu_manager.go:414] "containerMap: removing container" podUID="459e890f-044a-4949-a9b8-6a1f40394cd0" containerName="istio-proxy" Nov 03 07:28:51.781758 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.781745 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459e890f-044a-4949-a9b8-6a1f40394cd0" containerName="istio-proxy" Nov 03 07:28:51.783056 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.782814 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="702dd12a-8cf6-4daa-9530-8f85b6a8d01c" containerName="istio-proxy" Nov 03 07:28:51.783243 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.783228 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerName="istio-proxy" Nov 03 07:28:51.783333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.783323 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb42d5d-279b-4616-9b25-6d02bce7c1e3" containerName="istio-proxy" Nov 03 07:28:51.783421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.783412 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d13c1e-7bc8-4579-9dd7-80c31d00cad2" containerName="istio-proxy" Nov 03 07:28:51.783504 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.783494 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bfdadfa-4e8d-49ca-a1bf-4a3cd6a897b8" containerName="istio-proxy" Nov 03 07:28:51.783585 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.783576 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7f6705-eb50-49b1-bb3c-dcd0761f2c66" containerName="istio-proxy" Nov 03 07:28:51.783661 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.783652 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7ec919-d5c4-44e9-a498-82e604f21880" containerName="istio-proxy" Nov 03 07:28:51.783784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.783773 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="459e890f-044a-4949-a9b8-6a1f40394cd0" containerName="istio-proxy" Nov 03 07:28:51.783864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.783855 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0484f1-4f69-4550-a16f-1ccc33b0c0f4" containerName="istio-proxy" Nov 03 07:28:51.783940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.783931 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec563540-af7c-4fe7-8b7f-cc4be7102ab8" containerName="istio-proxy" Nov 03 07:28:51.790661 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.790642 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" Nov 03 07:28:51.809737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.809401 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf"] Nov 03 07:28:51.854752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.854700 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-5bdb10\"" Nov 03 07:28:51.854898 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.854849 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-5bdb10-dockercfg-n42kb\"" Nov 03 07:28:51.896991 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.877729 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltg9g\" (UniqueName: \"kubernetes.io/projected/090c672f-286d-4291-a953-3eddfdadbb15-kube-api-access-ltg9g\") pod \"payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf\" (UID: \"090c672f-286d-4291-a953-3eddfdadbb15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" Nov 03 07:28:51.896991 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.889917 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090c672f-286d-4291-a953-3eddfdadbb15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf\" (UID: \"090c672f-286d-4291-a953-3eddfdadbb15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" Nov 03 07:28:51.926581 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.913027 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2"] Nov 03 07:28:51.946745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.935970 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" Nov 03 07:28:51.975115 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.975030 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:28:51.987553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.978129 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:28:51.987553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.982992 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:28:52.033707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.993335 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltg9g\" (UniqueName: \"kubernetes.io/projected/090c672f-286d-4291-a953-3eddfdadbb15-kube-api-access-ltg9g\") pod \"payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf\" (UID: \"090c672f-286d-4291-a953-3eddfdadbb15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" Nov 03 07:28:52.033707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.993474 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090c672f-286d-4291-a953-3eddfdadbb15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf\" (UID: \"090c672f-286d-4291-a953-3eddfdadbb15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" Nov 03 07:28:52.033707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.993588 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjv47\" (UniqueName: \"kubernetes.io/projected/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-kube-api-access-vjv47\") pod \"payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2\" (UID: \"5ad61f40-0c74-4ea0-9dbe-1361ea562cba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" Nov 03 07:28:52.033707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.993622 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2\" (UID: \"5ad61f40-0c74-4ea0-9dbe-1361ea562cba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" Nov 03 07:28:52.033707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:51.994907 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090c672f-286d-4291-a953-3eddfdadbb15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf\" (UID: \"090c672f-286d-4291-a953-3eddfdadbb15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" Nov 03 07:28:52.095966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.095911 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2"] Nov 03 07:28:52.103792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.103758 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjv47\" (UniqueName: \"kubernetes.io/projected/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-kube-api-access-vjv47\") pod \"payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2\" (UID: \"5ad61f40-0c74-4ea0-9dbe-1361ea562cba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" Nov 03 07:28:52.104032 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.104014 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2\" (UID: \"5ad61f40-0c74-4ea0-9dbe-1361ea562cba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" Nov 03 07:28:52.104924 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.104896 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2\" (UID: \"5ad61f40-0c74-4ea0-9dbe-1361ea562cba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" Nov 03 07:28:52.140002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.131567 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltg9g\" (UniqueName: \"kubernetes.io/projected/090c672f-286d-4291-a953-3eddfdadbb15-kube-api-access-ltg9g\") pod \"payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf\" (UID: \"090c672f-286d-4291-a953-3eddfdadbb15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" Nov 03 07:28:52.140002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.137342 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" Nov 03 07:28:52.167910 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.167888 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjv47\" (UniqueName: \"kubernetes.io/projected/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-kube-api-access-vjv47\") pod \"payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2\" (UID: \"5ad61f40-0c74-4ea0-9dbe-1361ea562cba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" Nov 03 07:28:52.276881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.276845 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" Nov 03 07:28:52.536228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.536192 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf"] Nov 03 07:28:52.536459 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:28:52.536438 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090c672f_286d_4291_a953_3eddfdadbb15.slice/crio-62cc4b284abf9c478bdad8d32f6f73385ade90e5049c9ae7957e929351c0c23e WatchSource:0}: Error finding container 62cc4b284abf9c478bdad8d32f6f73385ade90e5049c9ae7957e929351c0c23e: Status 404 returned error can't find the container with id 62cc4b284abf9c478bdad8d32f6f73385ade90e5049c9ae7957e929351c0c23e Nov 03 07:28:52.682506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.682482 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2"] Nov 03 07:28:52.683834 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:28:52.683810 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad61f40_0c74_4ea0_9dbe_1361ea562cba.slice/crio-e89f41cf800d2c25a252a4a77d67e7b3e49363978430ac3f13ef087870ad0f9a WatchSource:0}: Error finding container e89f41cf800d2c25a252a4a77d67e7b3e49363978430ac3f13ef087870ad0f9a: Status 404 returned error can't find the container with id e89f41cf800d2c25a252a4a77d67e7b3e49363978430ac3f13ef087870ad0f9a Nov 03 07:28:52.714454 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.714433 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" event={"ID":"5ad61f40-0c74-4ea0-9dbe-1361ea562cba","Type":"ContainerStarted","Data":"e89f41cf800d2c25a252a4a77d67e7b3e49363978430ac3f13ef087870ad0f9a"} Nov 03 07:28:52.716115 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.716080 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" event={"ID":"090c672f-286d-4291-a953-3eddfdadbb15","Type":"ContainerStarted","Data":"81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a"} Nov 03 07:28:52.716203 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.716139 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" event={"ID":"090c672f-286d-4291-a953-3eddfdadbb15","Type":"ContainerStarted","Data":"62cc4b284abf9c478bdad8d32f6f73385ade90e5049c9ae7957e929351c0c23e"} Nov 03 07:28:52.836304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:52.836252 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" podStartSLOduration=1.836238357 podStartE2EDuration="1.836238357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:52.832882215 +0000 UTC m=+1788.025423962" watchObservedRunningTime="2026-11-03 07:28:52.836238357 +0000 UTC m=+1788.028780106" Nov 03 07:28:53.725517 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:53.725477 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" event={"ID":"5ad61f40-0c74-4ea0-9dbe-1361ea562cba","Type":"ContainerStarted","Data":"a5975f9cfe011d0c6dfb5a4b72393dc2f062a4919fcd9d55342fb0e357cccb89"} Nov 03 07:28:53.725981 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:53.725566 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" Nov 03 07:28:53.725981 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:53.725754 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" Nov 03 07:28:53.845615 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:53.845566 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" podStartSLOduration=2.845553282 podStartE2EDuration="2.845553282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:53.844166599 +0000 UTC m=+1789.036708349" watchObservedRunningTime="2026-11-03 07:28:53.845553282 +0000 UTC m=+1789.038095071" Nov 03 07:28:57.015965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.015385 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5"] Nov 03 07:28:57.034025 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.033994 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5"] Nov 03 07:28:57.034905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.034612 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" Nov 03 07:28:57.158467 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.158431 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769e14ab-8bb4-4734-8fb7-017a5697a095-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5\" (UID: \"769e14ab-8bb4-4734-8fb7-017a5697a095\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" Nov 03 07:28:57.158660 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.158612 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4cmm\" (UniqueName: \"kubernetes.io/projected/769e14ab-8bb4-4734-8fb7-017a5697a095-kube-api-access-k4cmm\") pod \"payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5\" (UID: \"769e14ab-8bb4-4734-8fb7-017a5697a095\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" Nov 03 07:28:57.179170 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.179143 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-09fd61-dockercfg-h282t\"" Nov 03 07:28:57.179170 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.179160 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-09fd61\"" Nov 03 07:28:57.182793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.182772 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25"] Nov 03 07:28:57.186942 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.186928 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" Nov 03 07:28:57.200489 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.200467 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25"] Nov 03 07:28:57.262691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.262656 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769e14ab-8bb4-4734-8fb7-017a5697a095-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5\" (UID: \"769e14ab-8bb4-4734-8fb7-017a5697a095\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" Nov 03 07:28:57.262925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.262742 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4hsw\" (UniqueName: \"kubernetes.io/projected/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-kube-api-access-m4hsw\") pod \"payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25\" (UID: \"d7c9c0b6-fff4-4390-85ee-8f244f021f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" Nov 03 07:28:57.263008 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.262967 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25\" (UID: \"d7c9c0b6-fff4-4390-85ee-8f244f021f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" Nov 03 07:28:57.263062 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.263042 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4cmm\" (UniqueName: \"kubernetes.io/projected/769e14ab-8bb4-4734-8fb7-017a5697a095-kube-api-access-k4cmm\") pod \"payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5\" (UID: \"769e14ab-8bb4-4734-8fb7-017a5697a095\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" Nov 03 07:28:57.263571 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.263546 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769e14ab-8bb4-4734-8fb7-017a5697a095-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5\" (UID: \"769e14ab-8bb4-4734-8fb7-017a5697a095\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" Nov 03 07:28:57.327687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.327657 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4cmm\" (UniqueName: \"kubernetes.io/projected/769e14ab-8bb4-4734-8fb7-017a5697a095-kube-api-access-k4cmm\") pod \"payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5\" (UID: \"769e14ab-8bb4-4734-8fb7-017a5697a095\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" Nov 03 07:28:57.364082 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.364055 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4hsw\" (UniqueName: \"kubernetes.io/projected/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-kube-api-access-m4hsw\") pod \"payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25\" (UID: \"d7c9c0b6-fff4-4390-85ee-8f244f021f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" Nov 03 07:28:57.364208 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.364170 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25\" (UID: \"d7c9c0b6-fff4-4390-85ee-8f244f021f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" Nov 03 07:28:57.364841 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.364823 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25\" (UID: \"d7c9c0b6-fff4-4390-85ee-8f244f021f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" Nov 03 07:28:57.426543 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.426513 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4hsw\" (UniqueName: \"kubernetes.io/projected/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-kube-api-access-m4hsw\") pod \"payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25\" (UID: \"d7c9c0b6-fff4-4390-85ee-8f244f021f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" Nov 03 07:28:57.434289 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.434271 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" Nov 03 07:28:57.506280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.506041 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" Nov 03 07:28:57.610771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.609613 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" Nov 03 07:28:57.856747 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:28:57.856699 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769e14ab_8bb4_4734_8fb7_017a5697a095.slice/crio-647e8c9c08bfcc4fce74671f800efe585a8d905ff40fdf36d3de43e09d264083 WatchSource:0}: Error finding container 647e8c9c08bfcc4fce74671f800efe585a8d905ff40fdf36d3de43e09d264083: Status 404 returned error can't find the container with id 647e8c9c08bfcc4fce74671f800efe585a8d905ff40fdf36d3de43e09d264083 Nov 03 07:28:57.859517 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.859492 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5"] Nov 03 07:28:57.900819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.900733 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25"] Nov 03 07:28:57.902756 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:28:57.902730 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c9c0b6_fff4_4390_85ee_8f244f021f6e.slice/crio-cbffde7ca8f0d4922d3d7fff572d7075056d2644abd751d6b36bc4e0b12808d0 WatchSource:0}: Error finding container cbffde7ca8f0d4922d3d7fff572d7075056d2644abd751d6b36bc4e0b12808d0: Status 404 returned error can't find the container with id cbffde7ca8f0d4922d3d7fff572d7075056d2644abd751d6b36bc4e0b12808d0 Nov 03 07:28:57.935658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.935627 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" event={"ID":"d7c9c0b6-fff4-4390-85ee-8f244f021f6e","Type":"ContainerStarted","Data":"cbffde7ca8f0d4922d3d7fff572d7075056d2644abd751d6b36bc4e0b12808d0"} Nov 03 07:28:57.937316 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:57.937288 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" event={"ID":"769e14ab-8bb4-4734-8fb7-017a5697a095","Type":"ContainerStarted","Data":"647e8c9c08bfcc4fce74671f800efe585a8d905ff40fdf36d3de43e09d264083"} Nov 03 07:28:58.946047 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:58.945999 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" event={"ID":"769e14ab-8bb4-4734-8fb7-017a5697a095","Type":"ContainerStarted","Data":"d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9"} Nov 03 07:28:58.946440 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:58.946211 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" Nov 03 07:28:58.947912 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:58.947886 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" event={"ID":"d7c9c0b6-fff4-4390-85ee-8f244f021f6e","Type":"ContainerStarted","Data":"df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69"} Nov 03 07:28:58.948049 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:58.948022 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" Nov 03 07:28:59.067040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:59.066994 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" podStartSLOduration=3.066980123 podStartE2EDuration="3.066980123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:59.064625764 +0000 UTC m=+1794.257167513" watchObservedRunningTime="2026-11-03 07:28:59.066980123 +0000 UTC m=+1794.259521873" Nov 03 07:28:59.196668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:28:59.196607 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" podStartSLOduration=2.196586061 podStartE2EDuration="2.196586061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:28:59.189447784 +0000 UTC m=+1794.381989535" watchObservedRunningTime="2026-11-03 07:28:59.196586061 +0000 UTC m=+1794.389127813" Nov 03 07:29:04.736317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:04.736287 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" Nov 03 07:29:04.736807 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:04.736356 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" Nov 03 07:29:05.156835 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:05.156801 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:29:05.158051 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:05.158018 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:29:08.774349 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:29:08.774311 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:29:09.963458 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:09.963424 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" Nov 03 07:29:09.964475 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:09.963512 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" Nov 03 07:29:24.118196 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.118161 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz"] Nov 03 07:29:24.124936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.124909 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.227503 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.227473 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-295fba7c\"/\"openshift-service-ca.crt\"" Nov 03 07:29:24.227701 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.227684 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-295fba7c\"/\"kube-root-ca.crt\"" Nov 03 07:29:24.227892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.227870 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-295fba7c\"/\"e2e-lifecycle-model-295fba7c-kserve-self-signed-certs\"" Nov 03 07:29:24.227960 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.227920 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-295fba7c\"/\"default-dockercfg-ss9hv\"" Nov 03 07:29:24.228673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.228655 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz"] Nov 03 07:29:24.334313 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.334259 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-home\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.334517 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.334415 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tls-certs\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.334517 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.334505 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tmp-dir\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.334668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.334650 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-model-cache\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.334796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.334703 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvkz4\" (UniqueName: \"kubernetes.io/projected/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-kube-api-access-dvkz4\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.334796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.334783 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-dshm\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.436479 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.436398 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tmp-dir\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.436620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.436530 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-model-cache\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.436620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.436567 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvkz4\" (UniqueName: \"kubernetes.io/projected/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-kube-api-access-dvkz4\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.436620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.436610 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-dshm\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.436837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.436813 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-home\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.437021 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.436863 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tmp-dir\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.437021 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.436924 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tls-certs\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.437021 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.436949 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-model-cache\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.437171 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.437111 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-home\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.439073 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.439048 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-dshm\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.439409 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.439392 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tls-certs\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.499384 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.499354 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvkz4\" (UniqueName: \"kubernetes.io/projected/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-kube-api-access-dvkz4\") pod \"e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:24.742526 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:24.742437 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:25.137324 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:25.137296 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz"] Nov 03 07:29:25.138036 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:29:25.138013 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5ba6870_1fdb_4bb3_999e_c4e9b8f200db.slice/crio-a4e6ad60d1c7ad3ec20f2ea112b8254a94e7f7f729e3783080dfd8428df74dca WatchSource:0}: Error finding container a4e6ad60d1c7ad3ec20f2ea112b8254a94e7f7f729e3783080dfd8428df74dca: Status 404 returned error can't find the container with id a4e6ad60d1c7ad3ec20f2ea112b8254a94e7f7f729e3783080dfd8428df74dca Nov 03 07:29:26.141628 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:26.141590 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" event={"ID":"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db","Type":"ContainerStarted","Data":"5f1a3ce551a74fd80c782abcd4df80e94f9bdaa624a39b49a7255339bf7a8c9d"} Nov 03 07:29:26.142059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:26.141639 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" event={"ID":"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db","Type":"ContainerStarted","Data":"a4e6ad60d1c7ad3ec20f2ea112b8254a94e7f7f729e3783080dfd8428df74dca"} Nov 03 07:29:26.263911 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:26.263862 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" podStartSLOduration=2.2638490239999998 podStartE2EDuration="2.263849024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:29:26.26289942 +0000 UTC m=+1821.455441172" watchObservedRunningTime="2026-11-03 07:29:26.263849024 +0000 UTC m=+1821.456390785" Nov 03 07:29:26.743203 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:26.743167 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:32.543333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.543298 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz"] Nov 03 07:29:32.548687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.548668 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:32.654787 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.654751 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x4r7\" (UniqueName: \"kubernetes.io/projected/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-kube-api-access-8x4r7\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:32.655158 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.655129 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:32.655770 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.655744 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:32.656647 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.656623 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:32.657318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.657295 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-5183ae-serving-cert\"" Nov 03 07:29:32.677629 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.677600 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz"] Nov 03 07:29:32.758287 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.758255 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:32.758467 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.758384 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4r7\" (UniqueName: \"kubernetes.io/projected/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-kube-api-access-8x4r7\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:32.758540 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.758475 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:32.758540 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.758522 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:32.758698 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:29:32.758616 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-serving-cert: secret "maas-api-e2e-shared-a-w6-5183ae-serving-cert" not found Nov 03 07:29:32.758698 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:29:32.758686 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-maas-api-tls podName:0d7f5b7f-0ac7-40d4-be81-cd09336f25b8 nodeName:}" failed. No retries permitted until 2026-11-03 07:29:33.258666083 +0000 UTC m=+1828.451207838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-maas-api-tls") pod "maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" (UID: "0d7f5b7f-0ac7-40d4-be81-cd09336f25b8") : secret "maas-api-e2e-shared-a-w6-5183ae-serving-cert" not found Nov 03 07:29:32.759261 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.759241 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:32.761168 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.761142 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:32.822338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:32.822312 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4r7\" (UniqueName: \"kubernetes.io/projected/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-kube-api-access-8x4r7\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:33.267309 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:33.267224 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:33.269878 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:33.269858 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:33.464015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:33.463979 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:33.848959 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:29:33.848924 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7f5b7f_0ac7_40d4_be81_cd09336f25b8.slice/crio-745cb7fe4b259ffcd7de00e6072090cafd186f0b13c6224a54d8491082b3aae0 WatchSource:0}: Error finding container 745cb7fe4b259ffcd7de00e6072090cafd186f0b13c6224a54d8491082b3aae0: Status 404 returned error can't find the container with id 745cb7fe4b259ffcd7de00e6072090cafd186f0b13c6224a54d8491082b3aae0 Nov 03 07:29:33.849385 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:33.849114 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz"] Nov 03 07:29:33.850658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:33.850635 4409 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:29:34.197931 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:34.197849 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" event={"ID":"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8","Type":"ContainerStarted","Data":"745cb7fe4b259ffcd7de00e6072090cafd186f0b13c6224a54d8491082b3aae0"} Nov 03 07:29:34.743408 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:34.743388 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:34.757185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:34.757165 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:35.207328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:35.207288 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" event={"ID":"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8","Type":"ContainerStarted","Data":"cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9"} Nov 03 07:29:35.207825 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:35.207467 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:35.219017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:35.218998 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:29:35.330668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:35.330619 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" podStartSLOduration=2.467358081 podStartE2EDuration="3.330606161s" totalImagesPullingTime="863.24808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:29:35.327574134 +0000 UTC m=+1830.520115884" watchObservedRunningTime="2026-11-03 07:29:35.330606161 +0000 UTC m=+1830.523147910" Nov 03 07:29:38.776911 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:29:38.776870 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:29:41.221438 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:41.221407 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:29:42.713363 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.713327 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q"] Nov 03 07:29:42.717843 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.717820 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:42.761371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.761326 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:42.761582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.761473 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01edfa30-1ac5-411a-a084-933f1a3578dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:42.761582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.761537 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n26k\" (UniqueName: \"kubernetes.io/projected/01edfa30-1ac5-411a-a084-933f1a3578dd-kube-api-access-8n26k\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:42.761705 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.761622 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:42.827469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.827433 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-acdd08-serving-cert\"" Nov 03 07:29:42.831797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.831768 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q"] Nov 03 07:29:42.863397 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.863354 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:42.863576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.863480 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01edfa30-1ac5-411a-a084-933f1a3578dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:42.863576 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:29:42.863530 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-serving-cert: secret "maas-api-e2e-shared-a-w3-acdd08-serving-cert" not found Nov 03 07:29:42.863576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.863552 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n26k\" (UniqueName: \"kubernetes.io/projected/01edfa30-1ac5-411a-a084-933f1a3578dd-kube-api-access-8n26k\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:42.863799 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:29:42.863621 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-maas-api-tls podName:01edfa30-1ac5-411a-a084-933f1a3578dd nodeName:}" failed. No retries permitted until 2026-11-03 07:29:43.363598805 +0000 UTC m=+1838.556140556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-maas-api-tls") pod "maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" (UID: "01edfa30-1ac5-411a-a084-933f1a3578dd") : secret "maas-api-e2e-shared-a-w3-acdd08-serving-cert" not found Nov 03 07:29:42.863799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.863658 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:42.864246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.864224 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01edfa30-1ac5-411a-a084-933f1a3578dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:42.866589 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.866564 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:42.932237 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:42.932199 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n26k\" (UniqueName: \"kubernetes.io/projected/01edfa30-1ac5-411a-a084-933f1a3578dd-kube-api-access-8n26k\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:43.398892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:43.398858 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:43.401452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:43.401429 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:43.632340 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:43.632301 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:44.019864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:44.019833 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q"] Nov 03 07:29:44.020553 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:29:44.020531 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01edfa30_1ac5_411a_a084_933f1a3578dd.slice/crio-7232451c08973376524fcb32dc07489e750d2433f33f07d758dd124f47823009 WatchSource:0}: Error finding container 7232451c08973376524fcb32dc07489e750d2433f33f07d758dd124f47823009: Status 404 returned error can't find the container with id 7232451c08973376524fcb32dc07489e750d2433f33f07d758dd124f47823009 Nov 03 07:29:44.300735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:44.300641 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" event={"ID":"01edfa30-1ac5-411a-a084-933f1a3578dd","Type":"ContainerStarted","Data":"7232451c08973376524fcb32dc07489e750d2433f33f07d758dd124f47823009"} Nov 03 07:29:45.309670 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:45.309634 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" event={"ID":"01edfa30-1ac5-411a-a084-933f1a3578dd","Type":"ContainerStarted","Data":"033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf"} Nov 03 07:29:45.310163 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:45.309773 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:45.431192 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:45.431130 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" podStartSLOduration=2.59669398 podStartE2EDuration="3.431115493s" totalImagesPullingTime="834.421513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:29:45.428413348 +0000 UTC m=+1840.620955100" watchObservedRunningTime="2026-11-03 07:29:45.431115493 +0000 UTC m=+1840.623657242" Nov 03 07:29:47.404734 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:47.404636 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4"] Nov 03 07:29:47.405181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:47.404856 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" podUID="d24c0370-2c02-4cc9-a456-60059748b1cb" containerName="maas-api" containerID="cri-o://b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e" gracePeriod=30 Nov 03 07:29:47.842191 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:47.841832 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:29:47.999510 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:47.999473 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-maas-api-tls\") pod \"d24c0370-2c02-4cc9-a456-60059748b1cb\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " Nov 03 07:29:47.999761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:47.999551 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24c0370-2c02-4cc9-a456-60059748b1cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d24c0370-2c02-4cc9-a456-60059748b1cb-openshift-service-ca\") pod \"d24c0370-2c02-4cc9-a456-60059748b1cb\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " Nov 03 07:29:47.999868 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:47.999765 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-metrics-certs\") pod \"d24c0370-2c02-4cc9-a456-60059748b1cb\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " Nov 03 07:29:47.999933 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:47.999874 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24c0370-2c02-4cc9-a456-60059748b1cb-kube-api-access-j788q\" (UniqueName: \"kubernetes.io/projected/d24c0370-2c02-4cc9-a456-60059748b1cb-kube-api-access-j788q\") pod \"d24c0370-2c02-4cc9-a456-60059748b1cb\" (UID: \"d24c0370-2c02-4cc9-a456-60059748b1cb\") " Nov 03 07:29:48.000427 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.000395 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24c0370-2c02-4cc9-a456-60059748b1cb-openshift-service-ca" pod "d24c0370-2c02-4cc9-a456-60059748b1cb" (UID: "d24c0370-2c02-4cc9-a456-60059748b1cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:29:48.000612 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.000593 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d24c0370-2c02-4cc9-a456-60059748b1cb-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:48.003087 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.003062 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-maas-api-tls" pod "d24c0370-2c02-4cc9-a456-60059748b1cb" (UID: "d24c0370-2c02-4cc9-a456-60059748b1cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:29:48.003857 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.003813 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-metrics-certs" pod "d24c0370-2c02-4cc9-a456-60059748b1cb" (UID: "d24c0370-2c02-4cc9-a456-60059748b1cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:29:48.004974 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.004946 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24c0370-2c02-4cc9-a456-60059748b1cb-kube-api-access-j788q" pod "d24c0370-2c02-4cc9-a456-60059748b1cb" (UID: "d24c0370-2c02-4cc9-a456-60059748b1cb"). InnerVolumeSpecName "kube-api-access-j788q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:29:48.102198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.102166 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:48.102198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.102194 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d24c0370-2c02-4cc9-a456-60059748b1cb-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:48.102198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.102205 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j788q\" (UniqueName: \"kubernetes.io/projected/d24c0370-2c02-4cc9-a456-60059748b1cb-kube-api-access-j788q\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:29:48.331654 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.331617 4409 generic.go:356] "Generic (PLEG): container finished" podID="d24c0370-2c02-4cc9-a456-60059748b1cb" containerID="b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e" exitCode=0 Nov 03 07:29:48.331850 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.331682 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" Nov 03 07:29:48.331850 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.331700 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" event={"ID":"d24c0370-2c02-4cc9-a456-60059748b1cb","Type":"ContainerDied","Data":"b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e"} Nov 03 07:29:48.331850 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.331821 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" event={"ID":"d24c0370-2c02-4cc9-a456-60059748b1cb","Type":"ContainerDied","Data":"18648f5d1d625931f8384aa0fe7f3295bf69066fe6db7e3a786e8c3c880aa1d5"} Nov 03 07:29:48.331850 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.331851 4409 scope.go:122] "RemoveContainer" containerID="b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e" Nov 03 07:29:48.342858 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.342792 4409 scope.go:122] "RemoveContainer" containerID="b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e" Nov 03 07:29:48.343072 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:29:48.343054 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e\": container with ID starting with b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e not found: ID does not exist" containerID="b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e" Nov 03 07:29:48.343131 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.343079 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e"} err="failed to get container status \"b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e\": rpc error: code = NotFound desc = could not find container \"b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e\": container with ID starting with b39c33d2106e9facb18c195788ffd376208cb67d089db2afe54e9e9f5cfda90e not found: ID does not exist" Nov 03 07:29:48.566229 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.565832 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4"] Nov 03 07:29:48.573337 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.573302 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4"] Nov 03 07:29:48.914457 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:48.914426 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24c0370-2c02-4cc9-a456-60059748b1cb" path="/var/lib/kubelet/pods/d24c0370-2c02-4cc9-a456-60059748b1cb/volumes" Nov 03 07:29:49.017525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.017495 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr"] Nov 03 07:29:49.018249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.018234 4409 cpu_manager.go:414] "containerMap: removing container" podUID="d24c0370-2c02-4cc9-a456-60059748b1cb" containerName="maas-api" Nov 03 07:29:49.018308 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.018251 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24c0370-2c02-4cc9-a456-60059748b1cb" containerName="maas-api" Nov 03 07:29:49.018348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.018341 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24c0370-2c02-4cc9-a456-60059748b1cb" containerName="maas-api" Nov 03 07:29:49.023212 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.023193 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.078181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.078159 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-5bdb10-serving-cert\"" Nov 03 07:29:49.082612 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.082585 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr"] Nov 03 07:29:49.127180 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.127153 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t784n\" (UniqueName: \"kubernetes.io/projected/a6938ea1-064f-40fe-8e6e-670a4a9615f4-kube-api-access-t784n\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.127301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.127211 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.127365 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.127343 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.127471 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.127455 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6938ea1-064f-40fe-8e6e-670a4a9615f4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.228351 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.228279 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t784n\" (UniqueName: \"kubernetes.io/projected/a6938ea1-064f-40fe-8e6e-670a4a9615f4-kube-api-access-t784n\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.228517 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.228351 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.228517 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.228402 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.228517 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.228457 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6938ea1-064f-40fe-8e6e-670a4a9615f4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.228517 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:29:49.228502 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-serving-cert: secret "maas-api-e2e-shared-b-w5-5bdb10-serving-cert" not found Nov 03 07:29:49.228779 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:29:49.228569 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-maas-api-tls podName:a6938ea1-064f-40fe-8e6e-670a4a9615f4 nodeName:}" failed. No retries permitted until 2026-11-03 07:29:49.72854825 +0000 UTC m=+1844.921089979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-maas-api-tls") pod "maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" (UID: "a6938ea1-064f-40fe-8e6e-670a4a9615f4") : secret "maas-api-e2e-shared-b-w5-5bdb10-serving-cert" not found Nov 03 07:29:49.229216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.229197 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6938ea1-064f-40fe-8e6e-670a4a9615f4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.231109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.231091 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.291511 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.291490 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t784n\" (UniqueName: \"kubernetes.io/projected/a6938ea1-064f-40fe-8e6e-670a4a9615f4-kube-api-access-t784n\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.736304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.736272 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.738892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.738868 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:49.939652 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:49.939617 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:50.373581 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:50.373498 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr"] Nov 03 07:29:50.419735 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:29:50.415034 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6938ea1_064f_40fe_8e6e_670a4a9615f4.slice/crio-39f5cb8b02451cb3da870bff7ac763834ae4b1404225ba8fcf8a8276e32386c6 WatchSource:0}: Error finding container 39f5cb8b02451cb3da870bff7ac763834ae4b1404225ba8fcf8a8276e32386c6: Status 404 returned error can't find the container with id 39f5cb8b02451cb3da870bff7ac763834ae4b1404225ba8fcf8a8276e32386c6 Nov 03 07:29:50.624668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:50.624553 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-399d09e5-546995d-zp6k4" podUID="d24c0370-2c02-4cc9-a456-60059748b1cb" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.107:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:29:51.321322 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:51.321299 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:29:51.401558 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:51.401530 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" event={"ID":"a6938ea1-064f-40fe-8e6e-670a4a9615f4","Type":"ContainerStarted","Data":"0d38589c4ae94ff76d0b1b3039bb5fce031197a3501b0030bbea4a6e70d10aaa"} Nov 03 07:29:51.401737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:51.401584 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" event={"ID":"a6938ea1-064f-40fe-8e6e-670a4a9615f4","Type":"ContainerStarted","Data":"39f5cb8b02451cb3da870bff7ac763834ae4b1404225ba8fcf8a8276e32386c6"} Nov 03 07:29:51.401737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:51.401604 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:51.579008 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:51.578921 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" podStartSLOduration=2.784735741 podStartE2EDuration="3.578902575s" totalImagesPullingTime="794.166834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:29:51.571559438 +0000 UTC m=+1846.764101191" watchObservedRunningTime="2026-11-03 07:29:51.578902575 +0000 UTC m=+1846.771444326" Nov 03 07:29:57.471739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:57.468131 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:29:59.328694 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:59.328662 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs"] Nov 03 07:29:59.329194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:29:59.328903 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" containerID="cri-o://8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb" gracePeriod=30 Nov 03 07:30:00.248790 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:00.248759 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc"] Nov 03 07:30:00.253216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:00.253195 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" Nov 03 07:30:00.255851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:00.255820 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6ph6\" (UniqueName: \"kubernetes.io/projected/a16f6418-5e62-4e46-8685-15c5dde69bda-kube-api-access-p6ph6\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc\" (UID: \"a16f6418-5e62-4e46-8685-15c5dde69bda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" Nov 03 07:30:00.256162 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:00.256142 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16f6418-5e62-4e46-8685-15c5dde69bda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc\" (UID: \"a16f6418-5e62-4e46-8685-15c5dde69bda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" Nov 03 07:30:00.312940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:00.312892 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc"] Nov 03 07:30:00.356052 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:00.356022 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8x99x\"" Nov 03 07:30:00.357699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:00.357670 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6ph6\" (UniqueName: \"kubernetes.io/projected/a16f6418-5e62-4e46-8685-15c5dde69bda-kube-api-access-p6ph6\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc\" (UID: \"a16f6418-5e62-4e46-8685-15c5dde69bda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" Nov 03 07:30:00.357874 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:00.357784 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16f6418-5e62-4e46-8685-15c5dde69bda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc\" (UID: \"a16f6418-5e62-4e46-8685-15c5dde69bda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" Nov 03 07:30:00.358558 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:00.358538 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16f6418-5e62-4e46-8685-15c5dde69bda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc\" (UID: \"a16f6418-5e62-4e46-8685-15c5dde69bda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" Nov 03 07:30:00.428772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:00.428694 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6ph6\" (UniqueName: \"kubernetes.io/projected/a16f6418-5e62-4e46-8685-15c5dde69bda-kube-api-access-p6ph6\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc\" (UID: \"a16f6418-5e62-4e46-8685-15c5dde69bda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" Nov 03 07:30:00.570871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:00.570843 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" Nov 03 07:30:01.108789 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.108656 4409 patch_prober.go:29] interesting pod/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:30:01.108789 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.108729 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:30:01.124495 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.124459 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st"] Nov 03 07:30:01.143632 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.134251 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st"] Nov 03 07:30:01.143632 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.134370 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.189283 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.189240 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-418f41-openshift-default-dockercfg-x2lv5\"" Nov 03 07:30:01.232290 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.232249 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-token\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.232476 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.232347 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-certs\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.232572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.232549 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-podinfo\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.232676 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.232658 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-data\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.232834 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.232706 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-socket\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.232834 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.232761 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv4vp\" (UniqueName: \"kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-kube-api-access-lv4vp\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.232834 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.232790 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-envoy\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.233001 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.232836 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-credential-socket\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.233001 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.232893 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istiod-ca-cert\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.336686 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.336642 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-podinfo\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.336882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.336771 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-data\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.336882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.336808 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-socket\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.336882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.336846 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv4vp\" (UniqueName: \"kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-kube-api-access-lv4vp\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.337275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.336883 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-envoy\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.337275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.336939 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-credential-socket\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.337275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.336994 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istiod-ca-cert\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.337275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.337047 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-token\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.337275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.337071 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-certs\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.358024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.355430 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-data\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.358024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.355703 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-socket\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.372228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.355963 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-credential-socket\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.372228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.369679 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istiod-ca-cert\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.373033 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.372981 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-certs\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.387023 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.386225 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-podinfo\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.393327 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.393277 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-envoy\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.414436 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.414400 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv4vp\" (UniqueName: \"kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-kube-api-access-lv4vp\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.435489 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.435466 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-token\") pod \"e2e-api-a-418f41-openshift-default-6b76655469-5f4st\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.464426 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.464399 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:01.493559 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.493334 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc"] Nov 03 07:30:01.497058 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:30:01.497030 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16f6418_5e62_4e46_8685_15c5dde69bda.slice/crio-2571859d6d6afa17770b136286a62e9a2a349a329bc216fa2c8ace45da203254 WatchSource:0}: Error finding container 2571859d6d6afa17770b136286a62e9a2a349a329bc216fa2c8ace45da203254: Status 404 returned error can't find the container with id 2571859d6d6afa17770b136286a62e9a2a349a329bc216fa2c8ace45da203254 Nov 03 07:30:01.694591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.694504 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" event={"ID":"a16f6418-5e62-4e46-8685-15c5dde69bda","Type":"ContainerStarted","Data":"2571859d6d6afa17770b136286a62e9a2a349a329bc216fa2c8ace45da203254"} Nov 03 07:30:01.865601 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.865570 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st"] Nov 03 07:30:01.865697 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:30:01.865620 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c819ba_88c2_4593_95fe_7ba9f7fff015.slice/crio-f8e170475cc9aee8df9d12982e20438e733834660660c885d6caad22a9cf6061 WatchSource:0}: Error finding container f8e170475cc9aee8df9d12982e20438e733834660660c885d6caad22a9cf6061: Status 404 returned error can't find the container with id f8e170475cc9aee8df9d12982e20438e733834660660c885d6caad22a9cf6061 Nov 03 07:30:01.869317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.869277 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:30:01.869414 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.869370 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:30:01.869455 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:01.869413 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:30:02.704135 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:02.704101 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" event={"ID":"d5c819ba-88c2-4593-95fe-7ba9f7fff015","Type":"ContainerStarted","Data":"8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352"} Nov 03 07:30:02.704600 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:02.704161 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" event={"ID":"d5c819ba-88c2-4593-95fe-7ba9f7fff015","Type":"ContainerStarted","Data":"f8e170475cc9aee8df9d12982e20438e733834660660c885d6caad22a9cf6061"} Nov 03 07:30:02.705797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:02.705770 4409 generic.go:356] "Generic (PLEG): container finished" podID="a16f6418-5e62-4e46-8685-15c5dde69bda" containerID="ba5a3fbff3832ed9d6e063af4f0087f3ef82493c8f7d73ee12ef28cf17fade21" exitCode=60 Nov 03 07:30:02.705949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:02.705854 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" event={"ID":"a16f6418-5e62-4e46-8685-15c5dde69bda","Type":"ContainerDied","Data":"ba5a3fbff3832ed9d6e063af4f0087f3ef82493c8f7d73ee12ef28cf17fade21"} Nov 03 07:30:02.706070 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:02.706056 4409 scope.go:122] "RemoveContainer" containerID="ba5a3fbff3832ed9d6e063af4f0087f3ef82493c8f7d73ee12ef28cf17fade21" Nov 03 07:30:02.829173 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:02.829129 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" podStartSLOduration=2.8291154819999997 podStartE2EDuration="2.829115482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:02.825973928 +0000 UTC m=+1858.018515680" watchObservedRunningTime="2026-11-03 07:30:02.829115482 +0000 UTC m=+1858.021657255" Nov 03 07:30:03.465228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:03.465182 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:03.471909 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:03.471878 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:03.714097 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:03.714064 4409 generic.go:356] "Generic (PLEG): container finished" podID="a16f6418-5e62-4e46-8685-15c5dde69bda" containerID="7f9284d19f385f15154ed1fba7ae99dc4692181ed6a2b78be5177cea2e137f25" exitCode=60 Nov 03 07:30:03.714577 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:03.714180 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" event={"ID":"a16f6418-5e62-4e46-8685-15c5dde69bda","Type":"ContainerDied","Data":"7f9284d19f385f15154ed1fba7ae99dc4692181ed6a2b78be5177cea2e137f25"} Nov 03 07:30:03.714577 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:03.714260 4409 scope.go:122] "RemoveContainer" containerID="ba5a3fbff3832ed9d6e063af4f0087f3ef82493c8f7d73ee12ef28cf17fade21" Nov 03 07:30:03.714577 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:03.714515 4409 scope.go:122] "RemoveContainer" containerID="7f9284d19f385f15154ed1fba7ae99dc4692181ed6a2b78be5177cea2e137f25" Nov 03 07:30:03.714784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:03.714654 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:03.714973 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:30:03.714948 4409 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc_odh-ai-gateway-infra(a16f6418-5e62-4e46-8685-15c5dde69bda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" Nov 03 07:30:03.716039 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:03.716024 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:30:04.627431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.627404 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:30:04.722828 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.722795 4409 generic.go:356] "Generic (PLEG): container finished" podID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerID="8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb" exitCode=0 Nov 03 07:30:04.723216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.722844 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" Nov 03 07:30:04.723216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.722872 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" event={"ID":"cd9f7361-77a0-4e49-93dc-301cd4c69c40","Type":"ContainerDied","Data":"8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb"} Nov 03 07:30:04.723216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.722925 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs" event={"ID":"cd9f7361-77a0-4e49-93dc-301cd4c69c40","Type":"ContainerDied","Data":"fe64403729823c20bfeca987494483e99de5b4ef90f90cddd8ee2b7f4cf0e9de"} Nov 03 07:30:04.723216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.722953 4409 scope.go:122] "RemoveContainer" containerID="8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb" Nov 03 07:30:04.725301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.725283 4409 scope.go:122] "RemoveContainer" containerID="7f9284d19f385f15154ed1fba7ae99dc4692181ed6a2b78be5177cea2e137f25" Nov 03 07:30:04.725651 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:30:04.725627 4409 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc_odh-ai-gateway-infra(a16f6418-5e62-4e46-8685-15c5dde69bda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" Nov 03 07:30:04.734320 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.734306 4409 scope.go:122] "RemoveContainer" containerID="cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1" Nov 03 07:30:04.736698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.736670 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-credential-socket\") pod \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " Nov 03 07:30:04.736829 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.736812 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-envoy\") pod \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " Nov 03 07:30:04.736912 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.736844 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-podinfo\") pod \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " Nov 03 07:30:04.736912 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.736905 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-token\" (UniqueName: \"kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-token\") pod \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " Nov 03 07:30:04.737022 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.736967 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-credential-socket" pod "cd9f7361-77a0-4e49-93dc-301cd4c69c40" (UID: "cd9f7361-77a0-4e49-93dc-301cd4c69c40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:30:04.737022 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.736978 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istiod-ca-cert\") pod \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " Nov 03 07:30:04.737140 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.737025 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-kube-api-access-98qnf\" (UniqueName: \"kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-kube-api-access-98qnf\") pod \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " Nov 03 07:30:04.737140 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.737083 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-socket\") pod \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " Nov 03 07:30:04.737397 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.737376 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-socket" pod "cd9f7361-77a0-4e49-93dc-301cd4c69c40" (UID: "cd9f7361-77a0-4e49-93dc-301cd4c69c40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:30:04.737475 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.737449 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istiod-ca-cert" pod "cd9f7361-77a0-4e49-93dc-301cd4c69c40" (UID: "cd9f7361-77a0-4e49-93dc-301cd4c69c40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:30:04.737588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.737487 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-certs\") pod \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " Nov 03 07:30:04.737588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.737565 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-data\") pod \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\" (UID: \"cd9f7361-77a0-4e49-93dc-301cd4c69c40\") " Nov 03 07:30:04.737831 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.737770 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-certs" pod "cd9f7361-77a0-4e49-93dc-301cd4c69c40" (UID: "cd9f7361-77a0-4e49-93dc-301cd4c69c40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:30:04.738050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.738028 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-data" pod "cd9f7361-77a0-4e49-93dc-301cd4c69c40" (UID: "cd9f7361-77a0-4e49-93dc-301cd4c69c40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:30:04.738356 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.738337 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:04.738587 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.738568 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:04.738682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.738600 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:04.738682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.738616 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:04.738682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.738631 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:04.739416 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.739391 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-token" pod "cd9f7361-77a0-4e49-93dc-301cd4c69c40" (UID: "cd9f7361-77a0-4e49-93dc-301cd4c69c40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:30:04.740824 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.740800 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-envoy" pod "cd9f7361-77a0-4e49-93dc-301cd4c69c40" (UID: "cd9f7361-77a0-4e49-93dc-301cd4c69c40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:30:04.741169 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.741149 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-podinfo" pod "cd9f7361-77a0-4e49-93dc-301cd4c69c40" (UID: "cd9f7361-77a0-4e49-93dc-301cd4c69c40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:30:04.742019 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.742000 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-kube-api-access-98qnf" pod "cd9f7361-77a0-4e49-93dc-301cd4c69c40" (UID: "cd9f7361-77a0-4e49-93dc-301cd4c69c40"). InnerVolumeSpecName "kube-api-access-98qnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:30:04.776473 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.776448 4409 scope.go:122] "RemoveContainer" containerID="8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb" Nov 03 07:30:04.776884 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:30:04.776853 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb\": container with ID starting with 8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb not found: ID does not exist" containerID="8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb" Nov 03 07:30:04.777017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.776894 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb"} err="failed to get container status \"8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb\": rpc error: code = NotFound desc = could not find container \"8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb\": container with ID starting with 8fba5a6c9eec48d867651c52abeaa43378126aea1e2203ee0d3c71af641d0ceb not found: ID does not exist" Nov 03 07:30:04.777017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.776917 4409 scope.go:122] "RemoveContainer" containerID="cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1" Nov 03 07:30:04.777227 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:30:04.777206 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1\": container with ID starting with cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1 not found: ID does not exist" containerID="cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1" Nov 03 07:30:04.777291 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.777237 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1"} err="failed to get container status \"cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1\": rpc error: code = NotFound desc = could not find container \"cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1\": container with ID starting with cae08d37e30c8a6bcf2f06a1470040bae6710bfe289951adf5319a9ab2eaa1e1 not found: ID does not exist" Nov 03 07:30:04.840706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.840678 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98qnf\" (UniqueName: \"kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-kube-api-access-98qnf\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:04.840706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.840727 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:04.840920 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.840752 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:04.840920 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:04.840766 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9f7361-77a0-4e49-93dc-301cd4c69c40-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:05.144246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:05.144207 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs"] Nov 03 07:30:05.147849 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:05.147828 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-399d09e5-openshift-default-85d5f99789-sqmqs"] Nov 03 07:30:06.916125 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:06.916097 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" path="/var/lib/kubelet/pods/cd9f7361-77a0-4e49-93dc-301cd4c69c40/volumes" Nov 03 07:30:08.776752 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:30:08.776677 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:30:09.593406 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.593370 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd"] Nov 03 07:30:09.594096 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.594079 4409 cpu_manager.go:414] "containerMap: removing container" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" Nov 03 07:30:09.594096 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.594097 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" Nov 03 07:30:09.594182 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.594122 4409 cpu_manager.go:414] "containerMap: removing container" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" Nov 03 07:30:09.594182 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.594127 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" Nov 03 07:30:09.594257 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.594240 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" Nov 03 07:30:09.594257 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.594251 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9f7361-77a0-4e49-93dc-301cd4c69c40" containerName="istio-proxy" Nov 03 07:30:09.599471 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.599453 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" Nov 03 07:30:09.704740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.667620 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd"] Nov 03 07:30:09.705820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.697683 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353e508a-fd20-4860-b256-ba30c29d4911-plugins-config-volume\") pod \"payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd\" (UID: \"353e508a-fd20-4860-b256-ba30c29d4911\") " pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" Nov 03 07:30:09.718738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.714291 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcbpq\" (UniqueName: \"kubernetes.io/projected/353e508a-fd20-4860-b256-ba30c29d4911-kube-api-access-lcbpq\") pod \"payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd\" (UID: \"353e508a-fd20-4860-b256-ba30c29d4911\") " pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" Nov 03 07:30:09.718738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.716092 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-418f41-dockercfg-rmt4s\"" Nov 03 07:30:09.718738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.716181 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-418f41\"" Nov 03 07:30:09.826737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.826683 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g"] Nov 03 07:30:09.871857 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.871685 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353e508a-fd20-4860-b256-ba30c29d4911-plugins-config-volume\") pod \"payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd\" (UID: \"353e508a-fd20-4860-b256-ba30c29d4911\") " pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" Nov 03 07:30:09.872000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.871926 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcbpq\" (UniqueName: \"kubernetes.io/projected/353e508a-fd20-4860-b256-ba30c29d4911-kube-api-access-lcbpq\") pod \"payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd\" (UID: \"353e508a-fd20-4860-b256-ba30c29d4911\") " pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" Nov 03 07:30:09.874661 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.874633 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353e508a-fd20-4860-b256-ba30c29d4911-plugins-config-volume\") pod \"payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd\" (UID: \"353e508a-fd20-4860-b256-ba30c29d4911\") " pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" Nov 03 07:30:09.875960 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.875937 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g"] Nov 03 07:30:09.876086 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.876019 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" Nov 03 07:30:09.937065 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.937039 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcbpq\" (UniqueName: \"kubernetes.io/projected/353e508a-fd20-4860-b256-ba30c29d4911-kube-api-access-lcbpq\") pod \"payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd\" (UID: \"353e508a-fd20-4860-b256-ba30c29d4911\") " pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" Nov 03 07:30:09.973007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.972979 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g\" (UID: \"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" Nov 03 07:30:09.973192 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:09.973172 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg6ls\" (UniqueName: \"kubernetes.io/projected/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-kube-api-access-jg6ls\") pod \"payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g\" (UID: \"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" Nov 03 07:30:10.074528 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.074502 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g\" (UID: \"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" Nov 03 07:30:10.074647 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.074634 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg6ls\" (UniqueName: \"kubernetes.io/projected/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-kube-api-access-jg6ls\") pod \"payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g\" (UID: \"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" Nov 03 07:30:10.075255 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.075236 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g\" (UID: \"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" Nov 03 07:30:10.137906 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.137853 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg6ls\" (UniqueName: \"kubernetes.io/projected/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-kube-api-access-jg6ls\") pod \"payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g\" (UID: \"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" Nov 03 07:30:10.191844 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.191824 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" Nov 03 07:30:10.216538 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.216512 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" Nov 03 07:30:10.575667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.575643 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g"] Nov 03 07:30:10.577555 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:30:10.577532 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a02b3e_c2e6_4ca0_af9e_a39ca37f63dc.slice/crio-7fc377b6885842e4c6cc2d12bbe466d6f192aa1e57e5a6984122ddfa34f18e01 WatchSource:0}: Error finding container 7fc377b6885842e4c6cc2d12bbe466d6f192aa1e57e5a6984122ddfa34f18e01: Status 404 returned error can't find the container with id 7fc377b6885842e4c6cc2d12bbe466d6f192aa1e57e5a6984122ddfa34f18e01 Nov 03 07:30:10.625370 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.625345 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd"] Nov 03 07:30:10.627016 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:30:10.626993 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353e508a_fd20_4860_b256_ba30c29d4911.slice/crio-a01fc3b4ea039dd419c13cecf95e727ba45989b15ada7f7dff53a9908738af89 WatchSource:0}: Error finding container a01fc3b4ea039dd419c13cecf95e727ba45989b15ada7f7dff53a9908738af89: Status 404 returned error can't find the container with id a01fc3b4ea039dd419c13cecf95e727ba45989b15ada7f7dff53a9908738af89 Nov 03 07:30:10.883867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.883773 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" event={"ID":"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc","Type":"ContainerStarted","Data":"404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078"} Nov 03 07:30:10.883867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.883835 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" event={"ID":"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc","Type":"ContainerStarted","Data":"7fc377b6885842e4c6cc2d12bbe466d6f192aa1e57e5a6984122ddfa34f18e01"} Nov 03 07:30:10.883867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.883860 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" Nov 03 07:30:10.885686 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.885663 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" event={"ID":"353e508a-fd20-4860-b256-ba30c29d4911","Type":"ContainerStarted","Data":"cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655"} Nov 03 07:30:10.885860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.885702 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" event={"ID":"353e508a-fd20-4860-b256-ba30c29d4911","Type":"ContainerStarted","Data":"a01fc3b4ea039dd419c13cecf95e727ba45989b15ada7f7dff53a9908738af89"} Nov 03 07:30:10.885860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:10.885741 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" Nov 03 07:30:11.001409 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:11.001072 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" podStartSLOduration=2.001027202 podStartE2EDuration="2.001027202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:11.000562068 +0000 UTC m=+1866.193103838" watchObservedRunningTime="2026-11-03 07:30:11.001027202 +0000 UTC m=+1866.193568956" Nov 03 07:30:11.127757 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:11.126371 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" podStartSLOduration=2.126351762 podStartE2EDuration="2.126351762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:11.118393291 +0000 UTC m=+1866.310935043" watchObservedRunningTime="2026-11-03 07:30:11.126351762 +0000 UTC m=+1866.318893515" Nov 03 07:30:13.906353 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:13.906324 4409 scope.go:122] "RemoveContainer" containerID="7f9284d19f385f15154ed1fba7ae99dc4692181ed6a2b78be5177cea2e137f25" Nov 03 07:30:14.931784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:14.931663 4409 generic.go:356] "Generic (PLEG): container finished" podID="a16f6418-5e62-4e46-8685-15c5dde69bda" containerID="3bca48809a029ca26057a02f60c4f42a9190360ff2d8c0cb265cf3459d177320" exitCode=60 Nov 03 07:30:14.931784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:14.931703 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" event={"ID":"a16f6418-5e62-4e46-8685-15c5dde69bda","Type":"ContainerDied","Data":"3bca48809a029ca26057a02f60c4f42a9190360ff2d8c0cb265cf3459d177320"} Nov 03 07:30:14.932361 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:14.931793 4409 scope.go:122] "RemoveContainer" containerID="7f9284d19f385f15154ed1fba7ae99dc4692181ed6a2b78be5177cea2e137f25" Nov 03 07:30:14.932361 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:14.932145 4409 scope.go:122] "RemoveContainer" containerID="3bca48809a029ca26057a02f60c4f42a9190360ff2d8c0cb265cf3459d177320" Nov 03 07:30:14.932471 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:30:14.932447 4409 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc_odh-ai-gateway-infra(a16f6418-5e62-4e46-8685-15c5dde69bda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" Nov 03 07:30:16.063087 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:16.063055 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc"] Nov 03 07:30:16.224914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:16.224892 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" Nov 03 07:30:16.254175 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:16.254146 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16f6418-5e62-4e46-8685-15c5dde69bda-kube-api-access-p6ph6\" (UniqueName: \"kubernetes.io/projected/a16f6418-5e62-4e46-8685-15c5dde69bda-kube-api-access-p6ph6\") pod \"a16f6418-5e62-4e46-8685-15c5dde69bda\" (UID: \"a16f6418-5e62-4e46-8685-15c5dde69bda\") " Nov 03 07:30:16.254306 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:16.254189 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16f6418-5e62-4e46-8685-15c5dde69bda-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16f6418-5e62-4e46-8685-15c5dde69bda-maas-api-service-ca\") pod \"a16f6418-5e62-4e46-8685-15c5dde69bda\" (UID: \"a16f6418-5e62-4e46-8685-15c5dde69bda\") " Nov 03 07:30:16.254594 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:16.254572 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16f6418-5e62-4e46-8685-15c5dde69bda-maas-api-service-ca" pod "a16f6418-5e62-4e46-8685-15c5dde69bda" (UID: "a16f6418-5e62-4e46-8685-15c5dde69bda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:30:16.254877 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:16.254862 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16f6418-5e62-4e46-8685-15c5dde69bda-maas-api-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:16.256492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:16.256469 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16f6418-5e62-4e46-8685-15c5dde69bda-kube-api-access-p6ph6" pod "a16f6418-5e62-4e46-8685-15c5dde69bda" (UID: "a16f6418-5e62-4e46-8685-15c5dde69bda"). InnerVolumeSpecName "kube-api-access-p6ph6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:30:16.356995 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:16.356927 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6ph6\" (UniqueName: \"kubernetes.io/projected/a16f6418-5e62-4e46-8685-15c5dde69bda-kube-api-access-p6ph6\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:30:16.948826 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:16.948796 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" event={"ID":"a16f6418-5e62-4e46-8685-15c5dde69bda","Type":"ContainerDied","Data":"2571859d6d6afa17770b136286a62e9a2a349a329bc216fa2c8ace45da203254"} Nov 03 07:30:16.948826 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:16.948815 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc" Nov 03 07:30:16.949042 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:16.948852 4409 scope.go:122] "RemoveContainer" containerID="3bca48809a029ca26057a02f60c4f42a9190360ff2d8c0cb265cf3459d177320" Nov 03 07:30:17.118331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:17.118297 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc"] Nov 03 07:30:17.167337 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:17.167306 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5bdb10-29894850-z2xxc"] Nov 03 07:30:18.914794 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:18.914766 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" path="/var/lib/kubelet/pods/a16f6418-5e62-4e46-8685-15c5dde69bda/volumes" Nov 03 07:30:19.736761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.736728 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd"] Nov 03 07:30:19.737459 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.737443 4409 cpu_manager.go:414] "containerMap: removing container" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" containerName="cleanup" Nov 03 07:30:19.737524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.737460 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" containerName="cleanup" Nov 03 07:30:19.737524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.737481 4409 cpu_manager.go:414] "containerMap: removing container" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" containerName="cleanup" Nov 03 07:30:19.737524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.737486 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" containerName="cleanup" Nov 03 07:30:19.737682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.737579 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" containerName="cleanup" Nov 03 07:30:19.737682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.737588 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" containerName="cleanup" Nov 03 07:30:19.737682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.737598 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" containerName="cleanup" Nov 03 07:30:19.742806 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.742779 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.797941 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.797919 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-295fba7c-openshift-default-dockercfg-6kjds\"" Nov 03 07:30:19.824424 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.824389 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.824583 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.824561 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.824656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.824637 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.824851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.824828 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.824935 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.824899 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrncg\" (UniqueName: \"kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-kube-api-access-qrncg\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.825003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.824959 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.825050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.825023 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.825115 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.825101 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.825198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.825184 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.850660 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.850630 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd"] Nov 03 07:30:19.926283 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.926257 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrncg\" (UniqueName: \"kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-kube-api-access-qrncg\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.926779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.926295 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.926779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.926326 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.926779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.926397 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.926779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.926436 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.926779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.926490 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.926779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.926582 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.926779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.926638 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.926779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.926763 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.927408 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.926794 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.927408 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.926881 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.927408 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.926951 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.927408 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.927200 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.927629 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.927611 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.929372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.929349 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:19.929555 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:19.929536 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:20.005055 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:20.004988 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrncg\" (UniqueName: \"kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-kube-api-access-qrncg\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:20.017061 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:20.017039 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:20.057867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:20.057840 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:20.500962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:20.500936 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd"] Nov 03 07:30:20.501843 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:30:20.501817 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b5a99e_526e_4d5e_afc4_41beb52f6baa.slice/crio-53bed7a8a015fa250df689cf399d39252cac015ad5823193196e00d057ff3b04 WatchSource:0}: Error finding container 53bed7a8a015fa250df689cf399d39252cac015ad5823193196e00d057ff3b04: Status 404 returned error can't find the container with id 53bed7a8a015fa250df689cf399d39252cac015ad5823193196e00d057ff3b04 Nov 03 07:30:20.504316 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:20.504281 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:30:20.504404 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:20.504349 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:30:20.504404 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:20.504377 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:30:20.982530 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:20.982496 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" event={"ID":"e0b5a99e-526e-4d5e-afc4-41beb52f6baa","Type":"ContainerStarted","Data":"8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce"} Nov 03 07:30:20.982997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:20.982553 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" event={"ID":"e0b5a99e-526e-4d5e-afc4-41beb52f6baa","Type":"ContainerStarted","Data":"53bed7a8a015fa250df689cf399d39252cac015ad5823193196e00d057ff3b04"} Nov 03 07:30:21.060183 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:21.060145 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:21.062500 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:21.062443 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:30:21.062689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:21.062510 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" podUID="e0b5a99e-526e-4d5e-afc4-41beb52f6baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 07:30:21.108727 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:21.108668 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" podStartSLOduration=2.108653329 podStartE2EDuration="2.108653329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:30:21.105918219 +0000 UTC m=+1876.298459968" watchObservedRunningTime="2026-11-03 07:30:21.108653329 +0000 UTC m=+1876.301195078" Nov 03 07:30:21.933088 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:21.931654 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" Nov 03 07:30:21.940131 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:21.940101 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" Nov 03 07:30:22.058197 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:22.058160 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:30:22.058639 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:22.058234 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" podUID="e0b5a99e-526e-4d5e-afc4-41beb52f6baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 07:30:23.059040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:23.059005 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:30:23.059422 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:23.059067 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" podUID="e0b5a99e-526e-4d5e-afc4-41beb52f6baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 07:30:24.059150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:24.059115 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:30:24.059756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:24.059703 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" podUID="e0b5a99e-526e-4d5e-afc4-41beb52f6baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 07:30:25.059023 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:25.058988 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:30:25.059210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:25.059035 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" podUID="e0b5a99e-526e-4d5e-afc4-41beb52f6baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 07:30:26.058993 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:26.058962 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 07:30:26.059193 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:26.059018 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" podUID="e0b5a99e-526e-4d5e-afc4-41beb52f6baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 07:30:27.062274 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:27.062242 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:27.062779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:27.062704 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:27.063642 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:27.063619 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:30:37.992779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:37.990808 4409 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Nov 03 07:30:38.014738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:38.003859 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:30:38.370748 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:38.367897 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 07:30:38.384896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:38.384847 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:30:38.614229 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:38.614186 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:30:38.917734 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:30:38.914607 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:30:45.674689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:45.674654 4409 generic.go:356] "Generic (PLEG): container finished" podID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerID="d78af7f55956bbafcc7891881249ab41125e112f5db8fc8fed77b969a750ebe1" exitCode=0 Nov 03 07:30:45.675241 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:45.674756 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" event={"ID":"5cc8eec3-9498-4244-afe5-456b9989e5f9","Type":"ContainerDied","Data":"d78af7f55956bbafcc7891881249ab41125e112f5db8fc8fed77b969a750ebe1"} Nov 03 07:30:45.675241 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:45.675204 4409 scope.go:122] "RemoveContainer" containerID="d78af7f55956bbafcc7891881249ab41125e112f5db8fc8fed77b969a750ebe1" Nov 03 07:30:45.676882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:45.676855 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:30:45.676971 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:45.676921 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:30:45.676971 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:45.676952 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:30:46.698465 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:46.698427 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" event={"ID":"5cc8eec3-9498-4244-afe5-456b9989e5f9","Type":"ContainerStarted","Data":"31a3001db283caa1629561eb98787e75ff0908ccde5bff9c5d18dc546eba96c0"} Nov 03 07:30:47.257681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:47.257648 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:30:47.257885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:47.257865 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 07:30:47.257943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:47.257910 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 07:30:48.258060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:48.258027 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 07:30:48.258475 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:48.258095 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 07:30:49.258073 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:49.258040 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 07:30:49.258487 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:49.258095 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 07:30:50.258070 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:50.258034 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 07:30:50.258243 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:50.258121 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 07:30:51.257889 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:51.257845 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 07:30:51.258062 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:51.257921 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 07:30:52.262514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:52.262481 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:30:52.262980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:52.262749 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:30:52.263474 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:30:52.263453 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:31:06.829570 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:06.829540 4409 scope.go:122] "RemoveContainer" containerID="b9c9dc5779bef400651a43d658fda0d98cd1696b1f4459e266f1a2e4f04ee668" Nov 03 07:31:08.776726 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:31:08.776673 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:31:09.319737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.319657 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv"] Nov 03 07:31:09.323739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.321045 4409 cpu_manager.go:414] "containerMap: removing container" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" containerName="cleanup" Nov 03 07:31:09.323739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.321072 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16f6418-5e62-4e46-8685-15c5dde69bda" containerName="cleanup" Nov 03 07:31:09.362733 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.362280 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" Nov 03 07:31:09.395837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.395785 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv"] Nov 03 07:31:09.535119 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.535084 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-32b500\"" Nov 03 07:31:09.567272 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.535685 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-32b500-dockercfg-gb67s\"" Nov 03 07:31:09.582865 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.568082 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv\" (UID: \"1d20a97a-9ef7-4e1a-b97e-5d420f609b59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" Nov 03 07:31:09.644200 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.644166 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbv4w\" (UniqueName: \"kubernetes.io/projected/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-kube-api-access-pbv4w\") pod \"payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv\" (UID: \"1d20a97a-9ef7-4e1a-b97e-5d420f609b59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" Nov 03 07:31:09.746581 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.746549 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv\" (UID: \"1d20a97a-9ef7-4e1a-b97e-5d420f609b59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" Nov 03 07:31:09.746761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.746658 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbv4w\" (UniqueName: \"kubernetes.io/projected/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-kube-api-access-pbv4w\") pod \"payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv\" (UID: \"1d20a97a-9ef7-4e1a-b97e-5d420f609b59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" Nov 03 07:31:09.747287 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.747267 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv\" (UID: \"1d20a97a-9ef7-4e1a-b97e-5d420f609b59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" Nov 03 07:31:09.856150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:09.855982 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbv4w\" (UniqueName: \"kubernetes.io/projected/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-kube-api-access-pbv4w\") pod \"payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv\" (UID: \"1d20a97a-9ef7-4e1a-b97e-5d420f609b59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" Nov 03 07:31:10.030643 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:10.030608 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" Nov 03 07:31:10.669635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:10.669607 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv"] Nov 03 07:31:10.670637 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:31:10.670611 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d20a97a_9ef7_4e1a_b97e_5d420f609b59.slice/crio-c0eb20193ffc6d2e4f39eefea5d1bff31b77f94f09adc74c457e7c18626a6190 WatchSource:0}: Error finding container c0eb20193ffc6d2e4f39eefea5d1bff31b77f94f09adc74c457e7c18626a6190: Status 404 returned error can't find the container with id c0eb20193ffc6d2e4f39eefea5d1bff31b77f94f09adc74c457e7c18626a6190 Nov 03 07:31:10.901983 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:10.901946 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" event={"ID":"1d20a97a-9ef7-4e1a-b97e-5d420f609b59","Type":"ContainerStarted","Data":"0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff"} Nov 03 07:31:10.902372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:10.902037 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" event={"ID":"1d20a97a-9ef7-4e1a-b97e-5d420f609b59","Type":"ContainerStarted","Data":"c0eb20193ffc6d2e4f39eefea5d1bff31b77f94f09adc74c457e7c18626a6190"} Nov 03 07:31:10.902372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:10.902072 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" Nov 03 07:31:11.019249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:11.019106 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" podStartSLOduration=2.019088617 podStartE2EDuration="2.019088617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:11.018791949 +0000 UTC m=+1926.211333699" watchObservedRunningTime="2026-11-03 07:31:11.019088617 +0000 UTC m=+1926.211630367" Nov 03 07:31:19.933937 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:19.933906 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w"] Nov 03 07:31:19.942834 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:19.942803 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.003548 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.003513 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w"] Nov 03 07:31:20.022246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.022209 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e62c887-4636-428c-aee4-10f5710eb996-openshift-service-ca\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.022457 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.022433 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-maas-api-tls\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.022547 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.022503 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqjd\" (UniqueName: \"kubernetes.io/projected/9e62c887-4636-428c-aee4-10f5710eb996-kube-api-access-ztqjd\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.022547 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.022533 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-metrics-certs\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.045038 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.044990 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-418f41-serving-cert\"" Nov 03 07:31:20.128234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.124864 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-maas-api-tls\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.128234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.124955 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqjd\" (UniqueName: \"kubernetes.io/projected/9e62c887-4636-428c-aee4-10f5710eb996-kube-api-access-ztqjd\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.128234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.124982 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-metrics-certs\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.128234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.125170 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e62c887-4636-428c-aee4-10f5710eb996-openshift-service-ca\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.128234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.126051 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e62c887-4636-428c-aee4-10f5710eb996-openshift-service-ca\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.128234 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:31:20.126162 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-serving-cert: secret "maas-api-e2e-api-a-418f41-serving-cert" not found Nov 03 07:31:20.128234 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:31:20.126228 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-maas-api-tls podName:9e62c887-4636-428c-aee4-10f5710eb996 nodeName:}" failed. No retries permitted until 2026-11-03 07:31:20.62620939 +0000 UTC m=+1935.818751133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-maas-api-tls") pod "maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" (UID: "9e62c887-4636-428c-aee4-10f5710eb996") : secret "maas-api-e2e-api-a-418f41-serving-cert" not found Nov 03 07:31:20.129773 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.129743 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-metrics-certs\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.194924 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.194886 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqjd\" (UniqueName: \"kubernetes.io/projected/9e62c887-4636-428c-aee4-10f5710eb996-kube-api-access-ztqjd\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.670410 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.669603 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-maas-api-tls\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.692506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.692471 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-maas-api-tls\") pod \"maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:20.873335 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:20.873055 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:21.287566 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:31:21.287532 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e62c887_4636_428c_aee4_10f5710eb996.slice/crio-1e6de5b558efd1129ea9835f197221610ab0a73d499470efe48dab9a3dc2c958 WatchSource:0}: Error finding container 1e6de5b558efd1129ea9835f197221610ab0a73d499470efe48dab9a3dc2c958: Status 404 returned error can't find the container with id 1e6de5b558efd1129ea9835f197221610ab0a73d499470efe48dab9a3dc2c958 Nov 03 07:31:21.289440 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:21.289406 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w"] Nov 03 07:31:21.936740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:21.930865 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" Nov 03 07:31:22.218782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:22.193376 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" event={"ID":"9e62c887-4636-428c-aee4-10f5710eb996","Type":"ContainerStarted","Data":"1e6de5b558efd1129ea9835f197221610ab0a73d499470efe48dab9a3dc2c958"} Nov 03 07:31:23.202970 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:23.202930 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" event={"ID":"9e62c887-4636-428c-aee4-10f5710eb996","Type":"ContainerStarted","Data":"82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c"} Nov 03 07:31:23.203422 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:23.203016 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:23.338646 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:23.338593 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" podStartSLOduration=3.292866177 podStartE2EDuration="4.338580686s" totalImagesPullingTime="1.045714509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:23.33507277 +0000 UTC m=+1938.527614522" watchObservedRunningTime="2026-11-03 07:31:23.338580686 +0000 UTC m=+1938.531122435" Nov 03 07:31:26.227913 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:26.227882 4409 generic.go:356] "Generic (PLEG): container finished" podID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerID="762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a" exitCode=0 Nov 03 07:31:26.228366 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:26.227929 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" event={"ID":"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91","Type":"ContainerDied","Data":"762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a"} Nov 03 07:31:26.228366 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:26.228296 4409 scope.go:122] "RemoveContainer" containerID="762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a" Nov 03 07:31:26.230237 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:26.230196 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:31:26.230376 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:26.230298 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:31:26.230376 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:26.230355 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:31:27.241850 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:27.236253 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:31:27.258813 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:27.254524 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" event={"ID":"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91","Type":"ContainerStarted","Data":"80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475"} Nov 03 07:31:28.096189 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:28.096150 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:31:28.097775 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:28.097749 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 07:31:28.097916 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:28.097800 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 07:31:29.095959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:29.095911 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 07:31:29.096390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:29.095980 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 07:31:29.216864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:29.216837 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:31:30.096199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:30.096166 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 07:31:30.096604 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:30.096228 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 07:31:31.095646 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:31.095612 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 07:31:31.095855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:31.095663 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 07:31:31.926138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:31.923551 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk"] Nov 03 07:31:31.964745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:31.959300 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" Nov 03 07:31:32.036739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.036666 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk"] Nov 03 07:31:32.244791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.229466 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk\" (UID: \"c96afb9c-3e74-479f-b1d2-f8dfce47b70e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" Nov 03 07:31:32.244791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.229640 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6hcb\" (UniqueName: \"kubernetes.io/projected/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-kube-api-access-v6hcb\") pod \"payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk\" (UID: \"c96afb9c-3e74-479f-b1d2-f8dfce47b70e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" Nov 03 07:31:32.250739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.247375 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-8e6dc7\"" Nov 03 07:31:32.311744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.297466 4409 patch_prober.go:29] interesting pod/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 07:31:32.311744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.297523 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 07:31:32.344751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.332841 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk\" (UID: \"c96afb9c-3e74-479f-b1d2-f8dfce47b70e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" Nov 03 07:31:32.345247 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.345216 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6hcb\" (UniqueName: \"kubernetes.io/projected/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-kube-api-access-v6hcb\") pod \"payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk\" (UID: \"c96afb9c-3e74-479f-b1d2-f8dfce47b70e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" Nov 03 07:31:32.355736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.339162 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk\" (UID: \"c96afb9c-3e74-479f-b1d2-f8dfce47b70e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" Nov 03 07:31:32.451022 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.448663 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8"] Nov 03 07:31:32.452355 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.452326 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6hcb\" (UniqueName: \"kubernetes.io/projected/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-kube-api-access-v6hcb\") pod \"payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk\" (UID: \"c96afb9c-3e74-479f-b1d2-f8dfce47b70e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" Nov 03 07:31:32.463978 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.463955 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" Nov 03 07:31:32.499086 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.499008 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8"] Nov 03 07:31:32.519170 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.519144 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-8e6dc7-dockercfg-j6tkf\"" Nov 03 07:31:32.596114 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.596087 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" Nov 03 07:31:32.649967 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.649930 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12866e2-730f-4719-b736-a337032baa17-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8\" (UID: \"a12866e2-730f-4719-b736-a337032baa17\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" Nov 03 07:31:32.650199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.650045 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8jvs\" (UniqueName: \"kubernetes.io/projected/a12866e2-730f-4719-b736-a337032baa17-kube-api-access-h8jvs\") pod \"payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8\" (UID: \"a12866e2-730f-4719-b736-a337032baa17\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" Nov 03 07:31:32.752420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.752332 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12866e2-730f-4719-b736-a337032baa17-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8\" (UID: \"a12866e2-730f-4719-b736-a337032baa17\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" Nov 03 07:31:32.752588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.752440 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8jvs\" (UniqueName: \"kubernetes.io/projected/a12866e2-730f-4719-b736-a337032baa17-kube-api-access-h8jvs\") pod \"payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8\" (UID: \"a12866e2-730f-4719-b736-a337032baa17\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" Nov 03 07:31:32.753277 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.753252 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12866e2-730f-4719-b736-a337032baa17-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8\" (UID: \"a12866e2-730f-4719-b736-a337032baa17\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" Nov 03 07:31:32.816164 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.816132 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8jvs\" (UniqueName: \"kubernetes.io/projected/a12866e2-730f-4719-b736-a337032baa17-kube-api-access-h8jvs\") pod \"payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8\" (UID: \"a12866e2-730f-4719-b736-a337032baa17\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" Nov 03 07:31:32.986472 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:32.986445 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk"] Nov 03 07:31:32.987012 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:31:32.986880 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96afb9c_3e74_479f_b1d2_f8dfce47b70e.slice/crio-d3972dcdaf26a6c3fbf7d7053cd5df5900795db356c39c436343460a390cf11c WatchSource:0}: Error finding container d3972dcdaf26a6c3fbf7d7053cd5df5900795db356c39c436343460a390cf11c: Status 404 returned error can't find the container with id d3972dcdaf26a6c3fbf7d7053cd5df5900795db356c39c436343460a390cf11c Nov 03 07:31:33.110637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:33.110609 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" Nov 03 07:31:33.131084 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:33.131060 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:31:33.132560 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:33.132486 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:31:33.133442 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:33.133403 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:31:33.460580 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:33.460534 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" event={"ID":"c96afb9c-3e74-479f-b1d2-f8dfce47b70e","Type":"ContainerStarted","Data":"a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d"} Nov 03 07:31:33.460797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:33.460599 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" event={"ID":"c96afb9c-3e74-479f-b1d2-f8dfce47b70e","Type":"ContainerStarted","Data":"d3972dcdaf26a6c3fbf7d7053cd5df5900795db356c39c436343460a390cf11c"} Nov 03 07:31:33.460797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:33.460630 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" Nov 03 07:31:33.505245 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:33.505201 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8"] Nov 03 07:31:33.505409 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:31:33.505379 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12866e2_730f_4719_b736_a337032baa17.slice/crio-d6937bcc832b17524db3d7cf1e4fc0ef6a4dcf72dd8b90fb6e1ec6cb5c67baae WatchSource:0}: Error finding container d6937bcc832b17524db3d7cf1e4fc0ef6a4dcf72dd8b90fb6e1ec6cb5c67baae: Status 404 returned error can't find the container with id d6937bcc832b17524db3d7cf1e4fc0ef6a4dcf72dd8b90fb6e1ec6cb5c67baae Nov 03 07:31:33.586345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:33.586301 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" podStartSLOduration=2.586287999 podStartE2EDuration="2.586287999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:33.582932866 +0000 UTC m=+1948.775474615" watchObservedRunningTime="2026-11-03 07:31:33.586287999 +0000 UTC m=+1948.778829748" Nov 03 07:31:34.518764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:34.517776 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" event={"ID":"a12866e2-730f-4719-b736-a337032baa17","Type":"ContainerStarted","Data":"163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461"} Nov 03 07:31:34.518764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:34.517846 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" event={"ID":"a12866e2-730f-4719-b736-a337032baa17","Type":"ContainerStarted","Data":"d6937bcc832b17524db3d7cf1e4fc0ef6a4dcf72dd8b90fb6e1ec6cb5c67baae"} Nov 03 07:31:34.518764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:34.517899 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" Nov 03 07:31:34.656163 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:34.655617 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" podStartSLOduration=2.655600949 podStartE2EDuration="2.655600949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:31:34.642124065 +0000 UTC m=+1949.834665816" watchObservedRunningTime="2026-11-03 07:31:34.655600949 +0000 UTC m=+1949.848142698" Nov 03 07:31:38.776694 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:31:38.776654 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:31:44.521334 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:44.521305 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" Nov 03 07:31:45.722055 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:45.722014 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" Nov 03 07:31:56.851953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:56.851920 4409 generic.go:356] "Generic (PLEG): container finished" podID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerID="8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352" exitCode=0 Nov 03 07:31:56.852377 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:56.851995 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" event={"ID":"d5c819ba-88c2-4593-95fe-7ba9f7fff015","Type":"ContainerDied","Data":"8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352"} Nov 03 07:31:56.852419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:56.852374 4409 scope.go:122] "RemoveContainer" containerID="8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352" Nov 03 07:31:56.854048 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:56.854013 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:31:56.854160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:56.854088 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:31:56.854160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:56.854124 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:31:57.861833 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:57.861792 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" event={"ID":"d5c819ba-88c2-4593-95fe-7ba9f7fff015","Type":"ContainerStarted","Data":"09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7"} Nov 03 07:31:58.474747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:58.464763 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:31:58.489933 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:58.489888 4409 patch_prober.go:29] interesting pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 07:31:58.490116 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:58.489961 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 07:31:59.474738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:59.464794 4409 patch_prober.go:29] interesting pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 07:31:59.474738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:31:59.464854 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 07:32:00.489785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:00.488091 4409 patch_prober.go:29] interesting pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 07:32:00.489785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:00.488163 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 07:32:01.489386 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:01.488812 4409 patch_prober.go:29] interesting pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 07:32:01.489386 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:01.488872 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 07:32:02.473740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:02.472150 4409 patch_prober.go:29] interesting pod/e2e-api-a-418f41-openshift-default-6b76655469-5f4st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 07:32:02.473740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:02.472245 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 07:32:03.472178 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:03.472146 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:32:03.472466 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:03.472443 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:32:03.473340 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:03.473319 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:32:08.775572 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:32:08.775533 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:32:21.960526 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:21.960494 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw"] Nov 03 07:32:26.960871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:26.960830 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" podUID="436d1679-cf9f-4ec0-a11a-efd0df1b6022" containerName="payload-processing" containerID="cri-o://c8d291fddae34ff874b05988035a3610576b5969f6eaf9f47601d43c97691b95" gracePeriod=25 Nov 03 07:32:27.107884 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:27.107846 4409 generic.go:356] "Generic (PLEG): container finished" podID="436d1679-cf9f-4ec0-a11a-efd0df1b6022" containerID="c8d291fddae34ff874b05988035a3610576b5969f6eaf9f47601d43c97691b95" exitCode=0 Nov 03 07:32:27.108029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:27.107920 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" event={"ID":"436d1679-cf9f-4ec0-a11a-efd0df1b6022","Type":"ContainerDied","Data":"c8d291fddae34ff874b05988035a3610576b5969f6eaf9f47601d43c97691b95"} Nov 03 07:32:27.235847 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:27.235789 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" Nov 03 07:32:27.393153 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:27.393122 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436d1679-cf9f-4ec0-a11a-efd0df1b6022-kube-api-access-t7bzg\" (UniqueName: \"kubernetes.io/projected/436d1679-cf9f-4ec0-a11a-efd0df1b6022-kube-api-access-t7bzg\") pod \"436d1679-cf9f-4ec0-a11a-efd0df1b6022\" (UID: \"436d1679-cf9f-4ec0-a11a-efd0df1b6022\") " Nov 03 07:32:27.393348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:27.393168 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436d1679-cf9f-4ec0-a11a-efd0df1b6022-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436d1679-cf9f-4ec0-a11a-efd0df1b6022-plugins-config-volume\") pod \"436d1679-cf9f-4ec0-a11a-efd0df1b6022\" (UID: \"436d1679-cf9f-4ec0-a11a-efd0df1b6022\") " Nov 03 07:32:27.393582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:27.393557 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436d1679-cf9f-4ec0-a11a-efd0df1b6022-plugins-config-volume" pod "436d1679-cf9f-4ec0-a11a-efd0df1b6022" (UID: "436d1679-cf9f-4ec0-a11a-efd0df1b6022"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:32:27.393810 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:27.393789 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436d1679-cf9f-4ec0-a11a-efd0df1b6022-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:32:27.396232 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:27.396201 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436d1679-cf9f-4ec0-a11a-efd0df1b6022-kube-api-access-t7bzg" pod "436d1679-cf9f-4ec0-a11a-efd0df1b6022" (UID: "436d1679-cf9f-4ec0-a11a-efd0df1b6022"). InnerVolumeSpecName "kube-api-access-t7bzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:32:27.495419 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:27.495314 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7bzg\" (UniqueName: \"kubernetes.io/projected/436d1679-cf9f-4ec0-a11a-efd0df1b6022-kube-api-access-t7bzg\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:32:28.116482 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:28.116454 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" Nov 03 07:32:28.116482 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:28.116478 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw" event={"ID":"436d1679-cf9f-4ec0-a11a-efd0df1b6022","Type":"ContainerDied","Data":"b073eb6ab42b3b569a9be99225db9997739136be5762b89248a110354e98a979"} Nov 03 07:32:28.117043 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:28.116537 4409 scope.go:122] "RemoveContainer" containerID="c8d291fddae34ff874b05988035a3610576b5969f6eaf9f47601d43c97691b95" Nov 03 07:32:28.287626 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:28.287582 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw"] Nov 03 07:32:28.337357 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:28.337319 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-8vscw"] Nov 03 07:32:28.914836 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:28.914800 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436d1679-cf9f-4ec0-a11a-efd0df1b6022" path="/var/lib/kubelet/pods/436d1679-cf9f-4ec0-a11a-efd0df1b6022/volumes" Nov 03 07:32:38.776536 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:32:38.776490 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:32:39.195940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.195901 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr"] Nov 03 07:32:39.197053 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.197033 4409 cpu_manager.go:414] "containerMap: removing container" podUID="436d1679-cf9f-4ec0-a11a-efd0df1b6022" containerName="payload-processing" Nov 03 07:32:39.197129 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.197055 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436d1679-cf9f-4ec0-a11a-efd0df1b6022" containerName="payload-processing" Nov 03 07:32:39.197188 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.197178 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="436d1679-cf9f-4ec0-a11a-efd0df1b6022" containerName="payload-processing" Nov 03 07:32:39.202540 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.202518 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.261113 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.261076 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr"] Nov 03 07:32:39.278141 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.278107 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.278320 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.278248 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc8vb\" (UniqueName: \"kubernetes.io/projected/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-kube-api-access-xc8vb\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.278387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.278352 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.278433 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.278412 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.304821 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.304780 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-1fa30f-serving-cert\"" Nov 03 07:32:39.380203 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.380172 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.380373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.380256 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc8vb\" (UniqueName: \"kubernetes.io/projected/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-kube-api-access-xc8vb\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.380373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.380285 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.380373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.380306 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.380495 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:32:39.380425 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-serving-cert: secret "maas-api-e2e-shared-b-w3-1fa30f-serving-cert" not found Nov 03 07:32:39.380541 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:32:39.380527 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-maas-api-tls podName:fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7 nodeName:}" failed. No retries permitted until 2026-11-03 07:32:39.880506928 +0000 UTC m=+2015.073048669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-maas-api-tls") pod "maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" (UID: "fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7") : secret "maas-api-e2e-shared-b-w3-1fa30f-serving-cert" not found Nov 03 07:32:39.381151 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.381124 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.382863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.382843 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.479763 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.479627 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc8vb\" (UniqueName: \"kubernetes.io/projected/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-kube-api-access-xc8vb\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.916355 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.916321 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:39.919027 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:39.919002 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:40.118541 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:40.118507 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:40.507311 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:40.507283 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr"] Nov 03 07:32:40.508408 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:32:40.508385 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb61af4a_ab5e_4ad1_85bc_b50a300bd9a7.slice/crio-7b6f3580fdcba37093bf678947d78078d522260afe91004cb6a654afab11e654 WatchSource:0}: Error finding container 7b6f3580fdcba37093bf678947d78078d522260afe91004cb6a654afab11e654: Status 404 returned error can't find the container with id 7b6f3580fdcba37093bf678947d78078d522260afe91004cb6a654afab11e654 Nov 03 07:32:41.219759 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:41.219679 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" event={"ID":"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7","Type":"ContainerStarted","Data":"7b6f3580fdcba37093bf678947d78078d522260afe91004cb6a654afab11e654"} Nov 03 07:32:42.229268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:42.229228 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" event={"ID":"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7","Type":"ContainerStarted","Data":"56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee"} Nov 03 07:32:42.229707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:42.229302 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:42.371635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:42.371568 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" podStartSLOduration=2.580270671 podStartE2EDuration="3.371544766s" totalImagesPullingTime="791.274095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:32:42.363249584 +0000 UTC m=+2017.555791360" watchObservedRunningTime="2026-11-03 07:32:42.371544766 +0000 UTC m=+2017.564086711" Nov 03 07:32:48.241638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:48.241540 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:32:59.847923 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:59.847891 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn"] Nov 03 07:32:59.852602 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:59.852581 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:32:59.931729 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:59.931682 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:32:59.931897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:59.931877 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:32:59.932073 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:59.932049 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296gr\" (UniqueName: \"kubernetes.io/projected/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-kube-api-access-296gr\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:32:59.932177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:59.932122 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:32:59.960739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:32:59.958000 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-8e6dc7-serving-cert\"" Nov 03 07:33:00.026745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:00.023388 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn"] Nov 03 07:33:00.077501 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:00.077376 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296gr\" (UniqueName: \"kubernetes.io/projected/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-kube-api-access-296gr\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:33:00.077953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:00.077912 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:33:00.078235 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:00.078217 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:33:00.078645 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:00.078622 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:33:00.087734 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:00.080458 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:33:00.088908 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:33:00.087984 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-serving-cert: secret "maas-api-e2e-shared-b-w2-8e6dc7-serving-cert" not found Nov 03 07:33:00.089173 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:33:00.089150 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-maas-api-tls podName:261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46 nodeName:}" failed. No retries permitted until 2026-11-03 07:33:00.589123422 +0000 UTC m=+2035.781665164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-maas-api-tls") pod "maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" (UID: "261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46") : secret "maas-api-e2e-shared-b-w2-8e6dc7-serving-cert" not found Nov 03 07:33:00.165838 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:00.163554 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:33:00.205226 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:00.189451 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-296gr\" (UniqueName: \"kubernetes.io/projected/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-kube-api-access-296gr\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:33:00.625064 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:00.625035 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:33:00.627843 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:00.627811 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:33:00.770430 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:00.770396 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:33:01.367334 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:01.367310 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn"] Nov 03 07:33:01.367980 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:33:01.367952 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261ffc0f_a6a3_4504_9cf4_ed54bb1e5e46.slice/crio-5feb3a5a4c884cbbe5ce450604e60ec99f899d818b5b8ddba36752bca51b3273 WatchSource:0}: Error finding container 5feb3a5a4c884cbbe5ce450604e60ec99f899d818b5b8ddba36752bca51b3273: Status 404 returned error can't find the container with id 5feb3a5a4c884cbbe5ce450604e60ec99f899d818b5b8ddba36752bca51b3273 Nov 03 07:33:01.433318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:01.433282 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" event={"ID":"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46","Type":"ContainerStarted","Data":"5feb3a5a4c884cbbe5ce450604e60ec99f899d818b5b8ddba36752bca51b3273"} Nov 03 07:33:02.441781 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:02.441749 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" event={"ID":"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46","Type":"ContainerStarted","Data":"0e8a7987fed62ed82d2381c31cc63ebb126e6edf8736a5e566c999828ea219eb"} Nov 03 07:33:02.442528 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:02.441981 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:33:02.570818 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:02.570762 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" podStartSLOduration=2.742910792 podStartE2EDuration="3.570744528s" totalImagesPullingTime="827.833736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:33:02.569079958 +0000 UTC m=+2037.761621710" watchObservedRunningTime="2026-11-03 07:33:02.570744528 +0000 UTC m=+2037.763286270" Nov 03 07:33:08.454269 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:08.454237 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:33:08.780962 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:33:08.780877 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:33:16.688165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:16.688129 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd"] Nov 03 07:33:16.806048 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:16.806008 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g"] Nov 03 07:33:21.688552 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:21.688501 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" podUID="353e508a-fd20-4860-b256-ba30c29d4911" containerName="payload-processing" containerID="cri-o://cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655" gracePeriod=25 Nov 03 07:33:21.807260 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:21.807207 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" podUID="92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc" containerName="payload-pre-processing" containerID="cri-o://404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078" gracePeriod=25 Nov 03 07:33:21.894597 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:33:21.894566 4409 logging.go:55] [core] [Channel #3486 SubChannel #3487]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.124:50052", ServerName: "10.133.0.124:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.124:50052: connect: connection refused" Nov 03 07:33:21.981501 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:21.981478 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" Nov 03 07:33:22.084911 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.084888 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" Nov 03 07:33:22.089104 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.089082 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353e508a-fd20-4860-b256-ba30c29d4911-kube-api-access-lcbpq\" (UniqueName: \"kubernetes.io/projected/353e508a-fd20-4860-b256-ba30c29d4911-kube-api-access-lcbpq\") pod \"353e508a-fd20-4860-b256-ba30c29d4911\" (UID: \"353e508a-fd20-4860-b256-ba30c29d4911\") " Nov 03 07:33:22.089328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.089315 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353e508a-fd20-4860-b256-ba30c29d4911-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353e508a-fd20-4860-b256-ba30c29d4911-plugins-config-volume\") pod \"353e508a-fd20-4860-b256-ba30c29d4911\" (UID: \"353e508a-fd20-4860-b256-ba30c29d4911\") " Nov 03 07:33:22.089699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.089644 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353e508a-fd20-4860-b256-ba30c29d4911-plugins-config-volume" pod "353e508a-fd20-4860-b256-ba30c29d4911" (UID: "353e508a-fd20-4860-b256-ba30c29d4911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:33:22.089699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.089809 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353e508a-fd20-4860-b256-ba30c29d4911-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:22.091318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.091292 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353e508a-fd20-4860-b256-ba30c29d4911-kube-api-access-lcbpq" pod "353e508a-fd20-4860-b256-ba30c29d4911" (UID: "353e508a-fd20-4860-b256-ba30c29d4911"). InnerVolumeSpecName "kube-api-access-lcbpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:33:22.191134 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.191099 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-kube-api-access-jg6ls\" (UniqueName: \"kubernetes.io/projected/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-kube-api-access-jg6ls\") pod \"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc\" (UID: \"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc\") " Nov 03 07:33:22.191293 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.191183 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-plugins-config-volume\") pod \"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc\" (UID: \"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc\") " Nov 03 07:33:22.191582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.191557 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-plugins-config-volume" pod "92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc" (UID: "92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:33:22.192054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.192035 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcbpq\" (UniqueName: \"kubernetes.io/projected/353e508a-fd20-4860-b256-ba30c29d4911-kube-api-access-lcbpq\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:22.192159 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.192056 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:22.193753 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.193696 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-kube-api-access-jg6ls" pod "92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc" (UID: "92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc"). InnerVolumeSpecName "kube-api-access-jg6ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:33:22.293692 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.293622 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg6ls\" (UniqueName: \"kubernetes.io/projected/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc-kube-api-access-jg6ls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:33:22.704474 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.704444 4409 generic.go:356] "Generic (PLEG): container finished" podID="353e508a-fd20-4860-b256-ba30c29d4911" containerID="cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655" exitCode=0 Nov 03 07:33:22.704945 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.704500 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" Nov 03 07:33:22.704945 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.704538 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" event={"ID":"353e508a-fd20-4860-b256-ba30c29d4911","Type":"ContainerDied","Data":"cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655"} Nov 03 07:33:22.704945 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.704600 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" event={"ID":"353e508a-fd20-4860-b256-ba30c29d4911","Type":"ContainerDied","Data":"a01fc3b4ea039dd419c13cecf95e727ba45989b15ada7f7dff53a9908738af89"} Nov 03 07:33:22.704945 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.704630 4409 scope.go:122] "RemoveContainer" containerID="cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655" Nov 03 07:33:22.706363 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.706340 4409 generic.go:356] "Generic (PLEG): container finished" podID="92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc" containerID="404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078" exitCode=0 Nov 03 07:33:22.706457 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.706365 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" event={"ID":"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc","Type":"ContainerDied","Data":"404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078"} Nov 03 07:33:22.706457 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.706396 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" Nov 03 07:33:22.706457 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.706410 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" event={"ID":"92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc","Type":"ContainerDied","Data":"7fc377b6885842e4c6cc2d12bbe466d6f192aa1e57e5a6984122ddfa34f18e01"} Nov 03 07:33:22.720770 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.720756 4409 scope.go:122] "RemoveContainer" containerID="cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655" Nov 03 07:33:22.721039 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:33:22.721020 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655\": container with ID starting with cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655 not found: ID does not exist" containerID="cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655" Nov 03 07:33:22.721078 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.721046 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655"} err="failed to get container status \"cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655\": rpc error: code = NotFound desc = could not find container \"cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655\": container with ID starting with cd3d97b475e1217cdbfb7b84f21ebdc5d73634b39b34b7e3423223f691de3655 not found: ID does not exist" Nov 03 07:33:22.721078 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.721061 4409 scope.go:122] "RemoveContainer" containerID="404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078" Nov 03 07:33:22.731792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.731771 4409 scope.go:122] "RemoveContainer" containerID="404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078" Nov 03 07:33:22.732074 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:33:22.732053 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078\": container with ID starting with 404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078 not found: ID does not exist" containerID="404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078" Nov 03 07:33:22.732132 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.732084 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078"} err="failed to get container status \"404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078\": rpc error: code = NotFound desc = could not find container \"404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078\": container with ID starting with 404c973c4ea5fe659e10abc83b0d96181deecd434c52a23e9a827391693f6078 not found: ID does not exist" Nov 03 07:33:22.874097 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.874063 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd"] Nov 03 07:33:22.894553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.894516 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g" podUID="92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.124:50052\" within 1s: context deadline exceeded" Nov 03 07:33:22.894686 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.894516 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd" podUID="353e508a-fd20-4860-b256-ba30c29d4911" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.123:50052\" within 1s: context deadline exceeded" Nov 03 07:33:22.894686 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:33:22.894594 4409 logging.go:55] [core] [Channel #3485 SubChannel #3488]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.123:50052", ServerName: "10.133.0.123:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.123:50052: operation was canceled" Nov 03 07:33:22.923771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.923735 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-418f41-b6f6b688c-z6xqd"] Nov 03 07:33:22.991181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.991123 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g"] Nov 03 07:33:22.998093 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:22.998066 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-418f41-c98644447-zpt8g"] Nov 03 07:33:24.914378 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:24.914343 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353e508a-fd20-4860-b256-ba30c29d4911" path="/var/lib/kubelet/pods/353e508a-fd20-4860-b256-ba30c29d4911/volumes" Nov 03 07:33:24.914816 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:33:24.914800 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc" path="/var/lib/kubelet/pods/92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc/volumes" Nov 03 07:33:38.777114 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:33:38.777064 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:34:05.215639 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:05.215604 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:34:05.220736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:05.220690 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:34:08.776200 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:08.776164 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:34:14.271048 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:14.271010 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc"] Nov 03 07:34:14.271600 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:14.271219 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" containerID="cri-o://44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6" gracePeriod=30 Nov 03 07:34:14.322180 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:14.322150 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg"] Nov 03 07:34:14.322420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:14.322396 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" containerID="cri-o://1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0" gracePeriod=30 Nov 03 07:34:14.326936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:14.326908 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m"] Nov 03 07:34:14.327109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:14.327087 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" podUID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" containerName="istio-proxy" containerID="cri-o://bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc" gracePeriod=30 Nov 03 07:34:14.370928 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:14.370896 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz"] Nov 03 07:34:14.371151 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:14.371126 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" containerID="cri-o://db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0" gracePeriod=30 Nov 03 07:34:14.376926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:14.376860 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw"] Nov 03 07:34:14.377170 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:14.377119 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" podUID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" containerName="istio-proxy" containerID="cri-o://87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e" gracePeriod=30 Nov 03 07:34:15.382377 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:15.382343 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl"] Nov 03 07:34:15.596633 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:15.596599 4409 patch_prober.go:29] interesting pod/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:34:15.596851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:15.596654 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" podUID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:34:18.151550 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:18.151462 4409 patch_prober.go:29] interesting pod/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:34:18.151550 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:18.151519 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" podUID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:34:19.401938 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:19.401909 4409 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482ac46d_f7b3_45f9_9923_4a8205c11e41.slice/crio-1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0.scope\": RecentStats: unable to find data in memory cache]" Nov 03 07:34:19.600063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.599727 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:34:19.733869 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.733800 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:34:19.739402 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.738981 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:34:19.751298 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.751270 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd5682e-484f-4582-b4aa-261a78c77d75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd5682e-484f-4582-b4aa-261a78c77d75-istiod-ca-cert\") pod \"9fd5682e-484f-4582-b4aa-261a78c77d75\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " Nov 03 07:34:19.751428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.751333 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-certs\") pod \"9fd5682e-484f-4582-b4aa-261a78c77d75\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " Nov 03 07:34:19.751428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.751390 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-token\" (UniqueName: \"kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-token\") pod \"9fd5682e-484f-4582-b4aa-261a78c77d75\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " Nov 03 07:34:19.751565 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.751436 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-envoy\") pod \"9fd5682e-484f-4582-b4aa-261a78c77d75\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " Nov 03 07:34:19.751565 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.751487 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-credential-socket\") pod \"9fd5682e-484f-4582-b4aa-261a78c77d75\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " Nov 03 07:34:19.751565 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.751523 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-data\") pod \"9fd5682e-484f-4582-b4aa-261a78c77d75\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " Nov 03 07:34:19.751757 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.751571 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-podinfo\") pod \"9fd5682e-484f-4582-b4aa-261a78c77d75\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " Nov 03 07:34:19.751757 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.751584 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-certs" pod "9fd5682e-484f-4582-b4aa-261a78c77d75" (UID: "9fd5682e-484f-4582-b4aa-261a78c77d75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.760318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.752384 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-kube-api-access-chmhg\" (UniqueName: \"kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-kube-api-access-chmhg\") pod \"9fd5682e-484f-4582-b4aa-261a78c77d75\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " Nov 03 07:34:19.760318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.752503 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-socket\") pod \"9fd5682e-484f-4582-b4aa-261a78c77d75\" (UID: \"9fd5682e-484f-4582-b4aa-261a78c77d75\") " Nov 03 07:34:19.760318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.753304 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.760318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.753571 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-socket" pod "9fd5682e-484f-4582-b4aa-261a78c77d75" (UID: "9fd5682e-484f-4582-b4aa-261a78c77d75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.760318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.754152 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd5682e-484f-4582-b4aa-261a78c77d75-istiod-ca-cert" pod "9fd5682e-484f-4582-b4aa-261a78c77d75" (UID: "9fd5682e-484f-4582-b4aa-261a78c77d75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:34:19.760318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.754522 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-podinfo" pod "9fd5682e-484f-4582-b4aa-261a78c77d75" (UID: "9fd5682e-484f-4582-b4aa-261a78c77d75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:34:19.760318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.754669 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-data" pod "9fd5682e-484f-4582-b4aa-261a78c77d75" (UID: "9fd5682e-484f-4582-b4aa-261a78c77d75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.760318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.754679 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-credential-socket" pod "9fd5682e-484f-4582-b4aa-261a78c77d75" (UID: "9fd5682e-484f-4582-b4aa-261a78c77d75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.760318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.756231 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-envoy" pod "9fd5682e-484f-4582-b4aa-261a78c77d75" (UID: "9fd5682e-484f-4582-b4aa-261a78c77d75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.760318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.756959 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-kube-api-access-chmhg" pod "9fd5682e-484f-4582-b4aa-261a78c77d75" (UID: "9fd5682e-484f-4582-b4aa-261a78c77d75"). InnerVolumeSpecName "kube-api-access-chmhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:19.760318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.759090 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-token" pod "9fd5682e-484f-4582-b4aa-261a78c77d75" (UID: "9fd5682e-484f-4582-b4aa-261a78c77d75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:19.808449 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.808428 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:34:19.811240 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.811211 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:34:19.854804 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.854745 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-socket\") pod \"482ac46d-f7b3-45f9-9923-4a8205c11e41\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " Nov 03 07:34:19.854804 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.854783 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-certs\") pod \"482ac46d-f7b3-45f9-9923-4a8205c11e41\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " Nov 03 07:34:19.854988 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.854837 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-token\" (UniqueName: \"kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-token\") pod \"9bfa7991-f674-43a6-82f0-174db2ff77fe\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " Nov 03 07:34:19.854988 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.854877 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-envoy\") pod \"482ac46d-f7b3-45f9-9923-4a8205c11e41\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " Nov 03 07:34:19.854988 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.854922 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-certs\") pod \"9bfa7991-f674-43a6-82f0-174db2ff77fe\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " Nov 03 07:34:19.854988 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.854951 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-podinfo\") pod \"482ac46d-f7b3-45f9-9923-4a8205c11e41\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " Nov 03 07:34:19.854988 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.854976 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-kube-api-access-24vm2\" (UniqueName: \"kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-kube-api-access-24vm2\") pod \"9bfa7991-f674-43a6-82f0-174db2ff77fe\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " Nov 03 07:34:19.855249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855024 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-podinfo\") pod \"9bfa7991-f674-43a6-82f0-174db2ff77fe\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " Nov 03 07:34:19.855249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855038 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-socket" pod "482ac46d-f7b3-45f9-9923-4a8205c11e41" (UID: "482ac46d-f7b3-45f9-9923-4a8205c11e41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.855249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855155 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-certs" pod "482ac46d-f7b3-45f9-9923-4a8205c11e41" (UID: "482ac46d-f7b3-45f9-9923-4a8205c11e41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.855249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855208 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482ac46d-f7b3-45f9-9923-4a8205c11e41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482ac46d-f7b3-45f9-9923-4a8205c11e41-istiod-ca-cert\") pod \"482ac46d-f7b3-45f9-9923-4a8205c11e41\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " Nov 03 07:34:19.855460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855257 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-envoy\") pod \"9bfa7991-f674-43a6-82f0-174db2ff77fe\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " Nov 03 07:34:19.855460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855294 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-credential-socket\") pod \"9bfa7991-f674-43a6-82f0-174db2ff77fe\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " Nov 03 07:34:19.855460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855353 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bfa7991-f674-43a6-82f0-174db2ff77fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bfa7991-f674-43a6-82f0-174db2ff77fe-istiod-ca-cert\") pod \"9bfa7991-f674-43a6-82f0-174db2ff77fe\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " Nov 03 07:34:19.855460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855392 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-credential-socket\") pod \"482ac46d-f7b3-45f9-9923-4a8205c11e41\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " Nov 03 07:34:19.855460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855421 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-kube-api-access-nw7wf\" (UniqueName: \"kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-kube-api-access-nw7wf\") pod \"482ac46d-f7b3-45f9-9923-4a8205c11e41\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " Nov 03 07:34:19.855760 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855514 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-socket\") pod \"9bfa7991-f674-43a6-82f0-174db2ff77fe\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " Nov 03 07:34:19.855760 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855617 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482ac46d-f7b3-45f9-9923-4a8205c11e41-istiod-ca-cert" pod "482ac46d-f7b3-45f9-9923-4a8205c11e41" (UID: "482ac46d-f7b3-45f9-9923-4a8205c11e41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:34:19.855760 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855626 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-data\") pod \"9bfa7991-f674-43a6-82f0-174db2ff77fe\" (UID: \"9bfa7991-f674-43a6-82f0-174db2ff77fe\") " Nov 03 07:34:19.855932 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.855913 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-data" pod "9bfa7991-f674-43a6-82f0-174db2ff77fe" (UID: "9bfa7991-f674-43a6-82f0-174db2ff77fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.856029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.856008 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-token\" (UniqueName: \"kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-token\") pod \"482ac46d-f7b3-45f9-9923-4a8205c11e41\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " Nov 03 07:34:19.856095 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.856067 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-data\") pod \"482ac46d-f7b3-45f9-9923-4a8205c11e41\" (UID: \"482ac46d-f7b3-45f9-9923-4a8205c11e41\") " Nov 03 07:34:19.856890 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.856864 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.857005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.856895 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.857005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.856911 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.857005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.856926 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chmhg\" (UniqueName: \"kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-kube-api-access-chmhg\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.857005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.856942 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.857005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.856957 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd5682e-484f-4582-b4aa-261a78c77d75-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.857005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.856971 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.857005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.856985 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.857005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.856999 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482ac46d-f7b3-45f9-9923-4a8205c11e41-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.857452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.857013 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.857452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.857030 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.857452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.857044 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd5682e-484f-4582-b4aa-261a78c77d75-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.857452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.857256 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-data" pod "482ac46d-f7b3-45f9-9923-4a8205c11e41" (UID: "482ac46d-f7b3-45f9-9923-4a8205c11e41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.858268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.857689 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-podinfo" pod "9bfa7991-f674-43a6-82f0-174db2ff77fe" (UID: "9bfa7991-f674-43a6-82f0-174db2ff77fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:34:19.858268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.857766 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-certs" pod "9bfa7991-f674-43a6-82f0-174db2ff77fe" (UID: "9bfa7991-f674-43a6-82f0-174db2ff77fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.858268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.857932 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-credential-socket" pod "482ac46d-f7b3-45f9-9923-4a8205c11e41" (UID: "482ac46d-f7b3-45f9-9923-4a8205c11e41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.858268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.857952 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-credential-socket" pod "9bfa7991-f674-43a6-82f0-174db2ff77fe" (UID: "9bfa7991-f674-43a6-82f0-174db2ff77fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.858268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.858055 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-kube-api-access-24vm2" pod "9bfa7991-f674-43a6-82f0-174db2ff77fe" (UID: "9bfa7991-f674-43a6-82f0-174db2ff77fe"). InnerVolumeSpecName "kube-api-access-24vm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:19.858268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.858188 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-socket" pod "9bfa7991-f674-43a6-82f0-174db2ff77fe" (UID: "9bfa7991-f674-43a6-82f0-174db2ff77fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.858268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.858236 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfa7991-f674-43a6-82f0-174db2ff77fe-istiod-ca-cert" pod "9bfa7991-f674-43a6-82f0-174db2ff77fe" (UID: "9bfa7991-f674-43a6-82f0-174db2ff77fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:34:19.858905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.858679 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-podinfo" pod "482ac46d-f7b3-45f9-9923-4a8205c11e41" (UID: "482ac46d-f7b3-45f9-9923-4a8205c11e41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:34:19.860145 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.860109 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-token" pod "482ac46d-f7b3-45f9-9923-4a8205c11e41" (UID: "482ac46d-f7b3-45f9-9923-4a8205c11e41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:19.860377 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.860359 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-envoy" pod "9bfa7991-f674-43a6-82f0-174db2ff77fe" (UID: "9bfa7991-f674-43a6-82f0-174db2ff77fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.860600 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.860581 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-envoy" pod "482ac46d-f7b3-45f9-9923-4a8205c11e41" (UID: "482ac46d-f7b3-45f9-9923-4a8205c11e41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.860600 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.860585 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-kube-api-access-nw7wf" pod "482ac46d-f7b3-45f9-9923-4a8205c11e41" (UID: "482ac46d-f7b3-45f9-9923-4a8205c11e41"). InnerVolumeSpecName "kube-api-access-nw7wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:19.860702 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.860684 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-token" pod "9bfa7991-f674-43a6-82f0-174db2ff77fe" (UID: "9bfa7991-f674-43a6-82f0-174db2ff77fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:19.957779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.957743 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-token\" (UniqueName: \"kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-token\") pod \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " Nov 03 07:34:19.957969 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.957840 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-credential-socket\") pod \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " Nov 03 07:34:19.957969 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.957888 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istiod-ca-cert\") pod \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " Nov 03 07:34:19.957969 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.957942 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-envoy\") pod \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " Nov 03 07:34:19.958146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.957986 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istiod-ca-cert\") pod \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " Nov 03 07:34:19.958146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958020 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-kube-api-access-mk52m\" (UniqueName: \"kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-kube-api-access-mk52m\") pod \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " Nov 03 07:34:19.958146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958059 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-envoy\") pod \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " Nov 03 07:34:19.958146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958099 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-podinfo\") pod \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " Nov 03 07:34:19.958146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958135 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-kube-api-access-tbfmt\" (UniqueName: \"kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-kube-api-access-tbfmt\") pod \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " Nov 03 07:34:19.958401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958293 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-socket\") pod \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " Nov 03 07:34:19.958401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958342 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-data\") pod \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " Nov 03 07:34:19.958401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958394 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-certs\") pod \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " Nov 03 07:34:19.958576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958430 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-socket\") pod \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " Nov 03 07:34:19.958576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958474 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-token\" (UniqueName: \"kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-token\") pod \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " Nov 03 07:34:19.958576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958530 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-certs\") pod \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\" (UID: \"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e\") " Nov 03 07:34:19.958576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958558 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-data\") pod \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " Nov 03 07:34:19.958576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958589 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-credential-socket\") pod \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " Nov 03 07:34:19.959022 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958619 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-podinfo\") pod \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\" (UID: \"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed\") " Nov 03 07:34:19.959022 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.958692 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istiod-ca-cert" pod "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" (UID: "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:34:19.959142 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959065 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-credential-socket" pod "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" (UID: "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.959474 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959448 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.959614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959479 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.959614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959497 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.959614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959512 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.959614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959526 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.959614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959540 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.959614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959556 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.959614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959570 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482ac46d-f7b3-45f9-9923-4a8205c11e41-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.959614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959585 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24vm2\" (UniqueName: \"kubernetes.io/projected/9bfa7991-f674-43a6-82f0-174db2ff77fe-kube-api-access-24vm2\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.959614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959595 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istiod-ca-cert" pod "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" (UID: "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:34:19.959614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959603 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.960150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959645 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.960150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959663 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.960150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959679 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bfa7991-f674-43a6-82f0-174db2ff77fe-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.960150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959693 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482ac46d-f7b3-45f9-9923-4a8205c11e41-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.960150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959725 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7wf\" (UniqueName: \"kubernetes.io/projected/482ac46d-f7b3-45f9-9923-4a8205c11e41-kube-api-access-nw7wf\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.960150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.959743 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bfa7991-f674-43a6-82f0-174db2ff77fe-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:19.961239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.960229 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-socket" pod "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" (UID: "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.961239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.960496 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-certs" pod "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" (UID: "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.961239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.960684 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-socket" pod "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" (UID: "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.961239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.960890 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-data" pod "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" (UID: "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.961239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.961073 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-certs" pod "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" (UID: "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.962206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.961244 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-credential-socket" pod "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" (UID: "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.962206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.961304 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-data" pod "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" (UID: "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.962517 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.962492 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-token" pod "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" (UID: "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:19.964206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.964176 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-token" pod "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" (UID: "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:19.964309 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.964279 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-envoy" pod "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" (UID: "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.964409 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.964385 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-podinfo" pod "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" (UID: "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:34:19.964758 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.964733 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-kube-api-access-mk52m" pod "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" (UID: "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e"). InnerVolumeSpecName "kube-api-access-mk52m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:19.965489 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.965451 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-kube-api-access-tbfmt" pod "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" (UID: "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed"). InnerVolumeSpecName "kube-api-access-tbfmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:19.965590 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.965548 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-envoy" pod "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" (UID: "74eb1078-e4b7-41f0-a4a8-5b8af714bf2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:19.966325 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:19.966300 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-podinfo" pod "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" (UID: "9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:34:20.061786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061758 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.061786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061783 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061808 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061842 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061857 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk52m\" (UniqueName: \"kubernetes.io/projected/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-kube-api-access-mk52m\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061872 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061887 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061901 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbfmt\" (UniqueName: \"kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-kube-api-access-tbfmt\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061916 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061931 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061946 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061960 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061973 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.061987 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.062000 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.062037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.062014 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.213836 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.213743 4409 generic.go:356] "Generic (PLEG): container finished" podID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerID="db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0" exitCode=0 Nov 03 07:34:20.213991 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.213831 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" event={"ID":"9bfa7991-f674-43a6-82f0-174db2ff77fe","Type":"ContainerDied","Data":"db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0"} Nov 03 07:34:20.213991 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.213858 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" Nov 03 07:34:20.213991 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.213907 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz" event={"ID":"9bfa7991-f674-43a6-82f0-174db2ff77fe","Type":"ContainerDied","Data":"683edc5bfd59efe33a8bd1547c8a7d755a345bd5ee1c5bf00e9bbc2a4c0b0844"} Nov 03 07:34:20.213991 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.213944 4409 scope.go:122] "RemoveContainer" containerID="db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0" Nov 03 07:34:20.215594 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.215552 4409 generic.go:356] "Generic (PLEG): container finished" podID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerID="44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6" exitCode=0 Nov 03 07:34:20.215907 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.215662 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" event={"ID":"9fd5682e-484f-4582-b4aa-261a78c77d75","Type":"ContainerDied","Data":"44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6"} Nov 03 07:34:20.215907 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.215675 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" Nov 03 07:34:20.215907 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.215700 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc" event={"ID":"9fd5682e-484f-4582-b4aa-261a78c77d75","Type":"ContainerDied","Data":"efd600313dda4733e7da34643957c7657d77fe366109850b611e258a59bcc004"} Nov 03 07:34:20.222087 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.222065 4409 generic.go:356] "Generic (PLEG): container finished" podID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" containerID="bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc" exitCode=0 Nov 03 07:34:20.222200 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.222162 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" Nov 03 07:34:20.222271 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.222229 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" event={"ID":"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e","Type":"ContainerDied","Data":"bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc"} Nov 03 07:34:20.222330 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.222274 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m" event={"ID":"74eb1078-e4b7-41f0-a4a8-5b8af714bf2e","Type":"ContainerDied","Data":"f529d09248dd5e8498b43f4700dfa3406774ee2f8f13e1a0fbf9d0d01f4429e1"} Nov 03 07:34:20.224240 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.224218 4409 generic.go:356] "Generic (PLEG): container finished" podID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" containerID="87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e" exitCode=0 Nov 03 07:34:20.224417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.224368 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" Nov 03 07:34:20.224824 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.224762 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" event={"ID":"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed","Type":"ContainerDied","Data":"87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e"} Nov 03 07:34:20.224926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.224846 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw" event={"ID":"9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed","Type":"ContainerDied","Data":"1b2a72e67eeae817b7c2401162348366d5c72a90da883ef18d9242508bba8532"} Nov 03 07:34:20.226946 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.226916 4409 scope.go:122] "RemoveContainer" containerID="db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0" Nov 03 07:34:20.227417 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:20.227386 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0\": container with ID starting with db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0 not found: ID does not exist" containerID="db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0" Nov 03 07:34:20.227507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.227425 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0"} err="failed to get container status \"db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0\": rpc error: code = NotFound desc = could not find container \"db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0\": container with ID starting with db8feb7ed91074f5c5f9304f80615b74befd6505d8dbf4bba4e8ac35dd03e9c0 not found: ID does not exist" Nov 03 07:34:20.227507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.227449 4409 scope.go:122] "RemoveContainer" containerID="44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6" Nov 03 07:34:20.227507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.227499 4409 generic.go:356] "Generic (PLEG): container finished" podID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerID="1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0" exitCode=0 Nov 03 07:34:20.227690 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.227556 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" Nov 03 07:34:20.227690 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.227569 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" event={"ID":"482ac46d-f7b3-45f9-9923-4a8205c11e41","Type":"ContainerDied","Data":"1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0"} Nov 03 07:34:20.227690 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.227606 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg" event={"ID":"482ac46d-f7b3-45f9-9923-4a8205c11e41","Type":"ContainerDied","Data":"842118be2cb97644a40aef5205d1bbdeaae74a7d82021ce10dbc7066e4f796fd"} Nov 03 07:34:20.239296 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.239282 4409 scope.go:122] "RemoveContainer" containerID="44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6" Nov 03 07:34:20.239575 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:20.239556 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6\": container with ID starting with 44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6 not found: ID does not exist" containerID="44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6" Nov 03 07:34:20.239680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.239581 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6"} err="failed to get container status \"44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6\": rpc error: code = NotFound desc = could not find container \"44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6\": container with ID starting with 44fe07b9e01c594bc2fcaf0739caa1427162a232402e4250024d9cc3508229e6 not found: ID does not exist" Nov 03 07:34:20.239680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.239599 4409 scope.go:122] "RemoveContainer" containerID="bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc" Nov 03 07:34:20.252510 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.252487 4409 scope.go:122] "RemoveContainer" containerID="bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc" Nov 03 07:34:20.252986 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:20.252966 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc\": container with ID starting with bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc not found: ID does not exist" containerID="bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc" Nov 03 07:34:20.253060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.252990 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc"} err="failed to get container status \"bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc\": rpc error: code = NotFound desc = could not find container \"bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc\": container with ID starting with bcb859fdb4fbe7957b761b8c71fbd1c7b59c2e52521b0d451d1e2c8aafa8bdcc not found: ID does not exist" Nov 03 07:34:20.253060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.253008 4409 scope.go:122] "RemoveContainer" containerID="87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e" Nov 03 07:34:20.276103 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.276073 4409 scope.go:122] "RemoveContainer" containerID="87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e" Nov 03 07:34:20.276491 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:20.276468 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e\": container with ID starting with 87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e not found: ID does not exist" containerID="87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e" Nov 03 07:34:20.276576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.276501 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e"} err="failed to get container status \"87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e\": rpc error: code = NotFound desc = could not find container \"87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e\": container with ID starting with 87ca018176f448a370dd36bb3bddac0ae84e1d1ebcb321014939c17402da3c6e not found: ID does not exist" Nov 03 07:34:20.276576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.276518 4409 scope.go:122] "RemoveContainer" containerID="1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0" Nov 03 07:34:20.288297 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.288273 4409 scope.go:122] "RemoveContainer" containerID="1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0" Nov 03 07:34:20.288672 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:20.288642 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0\": container with ID starting with 1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0 not found: ID does not exist" containerID="1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0" Nov 03 07:34:20.288819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.288682 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0"} err="failed to get container status \"1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0\": rpc error: code = NotFound desc = could not find container \"1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0\": container with ID starting with 1e4eecb0e055ea8bd33ae048182cd77e7db604cf99adcf09a1d136f15867d1e0 not found: ID does not exist" Nov 03 07:34:20.383692 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.383586 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" podUID="9027924d-f9a7-4a80-aff2-bd54b2cb04cf" containerName="payload-pre-processing" containerID="cri-o://616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f" gracePeriod=25 Nov 03 07:34:20.391614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.391588 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg"] Nov 03 07:34:20.396053 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.396026 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-lzsdg"] Nov 03 07:34:20.503454 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.503432 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz"] Nov 03 07:34:20.509280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.509247 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-g4lcz"] Nov 03 07:34:20.624154 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.624106 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m"] Nov 03 07:34:20.631657 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.631626 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-qbc7m"] Nov 03 07:34:20.678673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.678647 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" Nov 03 07:34:20.742559 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.742537 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw"] Nov 03 07:34:20.753650 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.753596 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-lcpdw"] Nov 03 07:34:20.772809 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.772786 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-kube-api-access-zzgpt\" (UniqueName: \"kubernetes.io/projected/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-kube-api-access-zzgpt\") pod \"9027924d-f9a7-4a80-aff2-bd54b2cb04cf\" (UID: \"9027924d-f9a7-4a80-aff2-bd54b2cb04cf\") " Nov 03 07:34:20.773001 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.772984 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-plugins-config-volume\") pod \"9027924d-f9a7-4a80-aff2-bd54b2cb04cf\" (UID: \"9027924d-f9a7-4a80-aff2-bd54b2cb04cf\") " Nov 03 07:34:20.773318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.773298 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-plugins-config-volume" pod "9027924d-f9a7-4a80-aff2-bd54b2cb04cf" (UID: "9027924d-f9a7-4a80-aff2-bd54b2cb04cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:34:20.773484 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.773452 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:20.775104 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.775082 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-kube-api-access-zzgpt" pod "9027924d-f9a7-4a80-aff2-bd54b2cb04cf" (UID: "9027924d-f9a7-4a80-aff2-bd54b2cb04cf"). InnerVolumeSpecName "kube-api-access-zzgpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:20.863717 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.863689 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc"] Nov 03 07:34:20.869961 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.869936 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9697f7-openshift-default-74d989748c-p9scc"] Nov 03 07:34:20.875145 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:20.875124 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzgpt\" (UniqueName: \"kubernetes.io/projected/9027924d-f9a7-4a80-aff2-bd54b2cb04cf-kube-api-access-zzgpt\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:21.099210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.096669 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" path="/var/lib/kubelet/pods/482ac46d-f7b3-45f9-9923-4a8205c11e41/volumes" Nov 03 07:34:21.099210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.097424 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" path="/var/lib/kubelet/pods/74eb1078-e4b7-41f0-a4a8-5b8af714bf2e/volumes" Nov 03 07:34:21.099210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.098135 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" path="/var/lib/kubelet/pods/9bfa7991-f674-43a6-82f0-174db2ff77fe/volumes" Nov 03 07:34:21.099480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.099281 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" path="/var/lib/kubelet/pods/9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed/volumes" Nov 03 07:34:21.100030 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.100004 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" path="/var/lib/kubelet/pods/9fd5682e-484f-4582-b4aa-261a78c77d75/volumes" Nov 03 07:34:21.237301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.237267 4409 generic.go:356] "Generic (PLEG): container finished" podID="9027924d-f9a7-4a80-aff2-bd54b2cb04cf" containerID="616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f" exitCode=0 Nov 03 07:34:21.237453 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.237325 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" Nov 03 07:34:21.237453 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.237357 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" event={"ID":"9027924d-f9a7-4a80-aff2-bd54b2cb04cf","Type":"ContainerDied","Data":"616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f"} Nov 03 07:34:21.237453 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.237413 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl" event={"ID":"9027924d-f9a7-4a80-aff2-bd54b2cb04cf","Type":"ContainerDied","Data":"09d3ea01d049a11f3ad822248874379bff369ca65fb7f35bb4447c311c508614"} Nov 03 07:34:21.237453 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.237438 4409 scope.go:122] "RemoveContainer" containerID="616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f" Nov 03 07:34:21.248339 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.248319 4409 scope.go:122] "RemoveContainer" containerID="616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f" Nov 03 07:34:21.248627 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:21.248603 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f\": container with ID starting with 616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f not found: ID does not exist" containerID="616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f" Nov 03 07:34:21.248697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.248638 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f"} err="failed to get container status \"616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f\": rpc error: code = NotFound desc = could not find container \"616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f\": container with ID starting with 616172ce730789eab36fa0c3a1d044db242a66f39a9b01492a731bd09674d82f not found: ID does not exist" Nov 03 07:34:21.405262 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.405195 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl"] Nov 03 07:34:21.410787 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:21.410764 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5183ae-86cf479dc-srwcl"] Nov 03 07:34:22.914678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:22.914640 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9027924d-f9a7-4a80-aff2-bd54b2cb04cf" path="/var/lib/kubelet/pods/9027924d-f9a7-4a80-aff2-bd54b2cb04cf/volumes" Nov 03 07:34:25.145184 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:25.145155 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9"] Nov 03 07:34:30.146215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:30.146172 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" podUID="51cec096-fc4b-4a5f-9b33-0c33c2a13b49" containerName="payload-pre-processing" containerID="cri-o://eb0218ae7ca183e93ae8319fcf98f8e69ec4ca27873192e6446d61247690603a" gracePeriod=25 Nov 03 07:34:30.320458 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:30.320428 4409 generic.go:356] "Generic (PLEG): container finished" podID="51cec096-fc4b-4a5f-9b33-0c33c2a13b49" containerID="eb0218ae7ca183e93ae8319fcf98f8e69ec4ca27873192e6446d61247690603a" exitCode=0 Nov 03 07:34:30.320614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:30.320500 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" event={"ID":"51cec096-fc4b-4a5f-9b33-0c33c2a13b49","Type":"ContainerDied","Data":"eb0218ae7ca183e93ae8319fcf98f8e69ec4ca27873192e6446d61247690603a"} Nov 03 07:34:30.526171 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:30.520379 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" Nov 03 07:34:30.611213 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:30.611180 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-kube-api-access-pjf6d\" (UniqueName: \"kubernetes.io/projected/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-kube-api-access-pjf6d\") pod \"51cec096-fc4b-4a5f-9b33-0c33c2a13b49\" (UID: \"51cec096-fc4b-4a5f-9b33-0c33c2a13b49\") " Nov 03 07:34:30.611357 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:30.611278 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-plugins-config-volume\") pod \"51cec096-fc4b-4a5f-9b33-0c33c2a13b49\" (UID: \"51cec096-fc4b-4a5f-9b33-0c33c2a13b49\") " Nov 03 07:34:30.611662 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:30.611636 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-plugins-config-volume" pod "51cec096-fc4b-4a5f-9b33-0c33c2a13b49" (UID: "51cec096-fc4b-4a5f-9b33-0c33c2a13b49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:34:30.612054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:30.612030 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:30.613592 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:30.613570 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-kube-api-access-pjf6d" pod "51cec096-fc4b-4a5f-9b33-0c33c2a13b49" (UID: "51cec096-fc4b-4a5f-9b33-0c33c2a13b49"). InnerVolumeSpecName "kube-api-access-pjf6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:30.713819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:30.713781 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjf6d\" (UniqueName: \"kubernetes.io/projected/51cec096-fc4b-4a5f-9b33-0c33c2a13b49-kube-api-access-pjf6d\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:31.330151 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:31.330120 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" event={"ID":"51cec096-fc4b-4a5f-9b33-0c33c2a13b49","Type":"ContainerDied","Data":"b79560d8bcac9ba3286e731e5238c51ddd0c2b59e532079105fc291fee6db976"} Nov 03 07:34:31.330611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:31.330174 4409 scope.go:122] "RemoveContainer" containerID="eb0218ae7ca183e93ae8319fcf98f8e69ec4ca27873192e6446d61247690603a" Nov 03 07:34:31.330611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:31.330292 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9" Nov 03 07:34:31.555836 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:31.553457 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9"] Nov 03 07:34:31.578736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:31.575309 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w"] Nov 03 07:34:31.578736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:31.575540 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" podUID="9e62c887-4636-428c-aee4-10f5710eb996" containerName="maas-api" containerID="cri-o://82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c" gracePeriod=30 Nov 03 07:34:31.652398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:31.614934 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-acdd08-65b8997c4c-fw6l9"] Nov 03 07:34:32.144798 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.144775 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:34:32.154046 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.154023 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e62c887-4636-428c-aee4-10f5710eb996-kube-api-access-ztqjd\" (UniqueName: \"kubernetes.io/projected/9e62c887-4636-428c-aee4-10f5710eb996-kube-api-access-ztqjd\") pod \"9e62c887-4636-428c-aee4-10f5710eb996\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " Nov 03 07:34:32.154153 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.154061 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-maas-api-tls\") pod \"9e62c887-4636-428c-aee4-10f5710eb996\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " Nov 03 07:34:32.154206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.154189 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e62c887-4636-428c-aee4-10f5710eb996-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e62c887-4636-428c-aee4-10f5710eb996-openshift-service-ca\") pod \"9e62c887-4636-428c-aee4-10f5710eb996\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " Nov 03 07:34:32.154240 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.154217 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-metrics-certs\") pod \"9e62c887-4636-428c-aee4-10f5710eb996\" (UID: \"9e62c887-4636-428c-aee4-10f5710eb996\") " Nov 03 07:34:32.154532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.154500 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e62c887-4636-428c-aee4-10f5710eb996-openshift-service-ca" pod "9e62c887-4636-428c-aee4-10f5710eb996" (UID: "9e62c887-4636-428c-aee4-10f5710eb996"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:34:32.154891 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.154868 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e62c887-4636-428c-aee4-10f5710eb996-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:32.156291 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.156270 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-maas-api-tls" pod "9e62c887-4636-428c-aee4-10f5710eb996" (UID: "9e62c887-4636-428c-aee4-10f5710eb996"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:34:32.156376 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.156288 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e62c887-4636-428c-aee4-10f5710eb996-kube-api-access-ztqjd" pod "9e62c887-4636-428c-aee4-10f5710eb996" (UID: "9e62c887-4636-428c-aee4-10f5710eb996"). InnerVolumeSpecName "kube-api-access-ztqjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:32.156602 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.156574 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-metrics-certs" pod "9e62c887-4636-428c-aee4-10f5710eb996" (UID: "9e62c887-4636-428c-aee4-10f5710eb996"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:34:32.256469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.256399 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:32.256469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.256429 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqjd\" (UniqueName: \"kubernetes.io/projected/9e62c887-4636-428c-aee4-10f5710eb996-kube-api-access-ztqjd\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:32.256469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.256443 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e62c887-4636-428c-aee4-10f5710eb996-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:32.339929 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.339898 4409 generic.go:356] "Generic (PLEG): container finished" podID="9e62c887-4636-428c-aee4-10f5710eb996" containerID="82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c" exitCode=0 Nov 03 07:34:32.340331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.339942 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" Nov 03 07:34:32.340331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.339984 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" event={"ID":"9e62c887-4636-428c-aee4-10f5710eb996","Type":"ContainerDied","Data":"82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c"} Nov 03 07:34:32.340331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.340031 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w" event={"ID":"9e62c887-4636-428c-aee4-10f5710eb996","Type":"ContainerDied","Data":"1e6de5b558efd1129ea9835f197221610ab0a73d499470efe48dab9a3dc2c958"} Nov 03 07:34:32.340331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.340051 4409 scope.go:122] "RemoveContainer" containerID="82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c" Nov 03 07:34:32.351530 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.351516 4409 scope.go:122] "RemoveContainer" containerID="82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c" Nov 03 07:34:32.351819 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:32.351799 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c\": container with ID starting with 82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c not found: ID does not exist" containerID="82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c" Nov 03 07:34:32.351898 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.351830 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c"} err="failed to get container status \"82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c\": rpc error: code = NotFound desc = could not find container \"82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c\": container with ID starting with 82e857fa90dc660dafaf6cf4e8825649ecb34c444670c958de0cf7ebdfe08f3c not found: ID does not exist" Nov 03 07:34:32.546789 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.545764 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w"] Nov 03 07:34:32.572836 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.572797 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-418f41-849d4c6dbc-glx5w"] Nov 03 07:34:32.922444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.922410 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51cec096-fc4b-4a5f-9b33-0c33c2a13b49" path="/var/lib/kubelet/pods/51cec096-fc4b-4a5f-9b33-0c33c2a13b49/volumes" Nov 03 07:34:32.922967 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:32.922947 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e62c887-4636-428c-aee4-10f5710eb996" path="/var/lib/kubelet/pods/9e62c887-4636-428c-aee4-10f5710eb996/volumes" Nov 03 07:34:34.879345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:34.879313 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5"] Nov 03 07:34:34.997581 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:34.997551 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25"] Nov 03 07:34:38.456147 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456112 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g"] Nov 03 07:34:38.456824 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456810 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9027924d-f9a7-4a80-aff2-bd54b2cb04cf" containerName="payload-pre-processing" Nov 03 07:34:38.456892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456825 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9027924d-f9a7-4a80-aff2-bd54b2cb04cf" containerName="payload-pre-processing" Nov 03 07:34:38.456892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456862 4409 cpu_manager.go:414] "containerMap: removing container" podUID="353e508a-fd20-4860-b256-ba30c29d4911" containerName="payload-processing" Nov 03 07:34:38.456892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456870 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353e508a-fd20-4860-b256-ba30c29d4911" containerName="payload-processing" Nov 03 07:34:38.456892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456885 4409 cpu_manager.go:414] "containerMap: removing container" podUID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" containerName="istio-proxy" Nov 03 07:34:38.456892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456890 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" containerName="istio-proxy" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456901 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9e62c887-4636-428c-aee4-10f5710eb996" containerName="maas-api" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456906 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e62c887-4636-428c-aee4-10f5710eb996" containerName="maas-api" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456917 4409 cpu_manager.go:414] "containerMap: removing container" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456922 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456935 4409 cpu_manager.go:414] "containerMap: removing container" podUID="51cec096-fc4b-4a5f-9b33-0c33c2a13b49" containerName="payload-pre-processing" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456939 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cec096-fc4b-4a5f-9b33-0c33c2a13b49" containerName="payload-pre-processing" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456945 4409 cpu_manager.go:414] "containerMap: removing container" podUID="92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc" containerName="payload-pre-processing" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456949 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc" containerName="payload-pre-processing" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456962 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456966 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456973 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456977 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456989 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" containerName="istio-proxy" Nov 03 07:34:38.457040 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.456993 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" containerName="istio-proxy" Nov 03 07:34:38.457428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.457089 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="482ac46d-f7b3-45f9-9923-4a8205c11e41" containerName="istio-proxy" Nov 03 07:34:38.457428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.457098 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cec096-fc4b-4a5f-9b33-0c33c2a13b49" containerName="payload-pre-processing" Nov 03 07:34:38.457428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.457107 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a02b3e-c2e6-4ca0-af9e-a39ca37f63dc" containerName="payload-pre-processing" Nov 03 07:34:38.457428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.457113 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd5682e-484f-4582-b4aa-261a78c77d75" containerName="istio-proxy" Nov 03 07:34:38.457428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.457118 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfa7991-f674-43a6-82f0-174db2ff77fe" containerName="istio-proxy" Nov 03 07:34:38.457428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.457124 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3c1f7d-b9c6-4725-a926-1eb85ecaf7ed" containerName="istio-proxy" Nov 03 07:34:38.457428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.457131 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9027924d-f9a7-4a80-aff2-bd54b2cb04cf" containerName="payload-pre-processing" Nov 03 07:34:38.457428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.457139 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="353e508a-fd20-4860-b256-ba30c29d4911" containerName="payload-processing" Nov 03 07:34:38.457428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.457144 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="74eb1078-e4b7-41f0-a4a8-5b8af714bf2e" containerName="istio-proxy" Nov 03 07:34:38.457428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.457150 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e62c887-4636-428c-aee4-10f5710eb996" containerName="maas-api" Nov 03 07:34:38.462273 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.462251 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:38.518622 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.518587 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g"] Nov 03 07:34:38.551050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.551004 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-metrics-certs\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:38.551245 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.551221 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e84d830-9ae7-41f5-a105-068e1d316171-openshift-service-ca\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:38.551302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.551276 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msvlc\" (UniqueName: \"kubernetes.io/projected/7e84d830-9ae7-41f5-a105-068e1d316171-kube-api-access-msvlc\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:38.551404 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.551380 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-maas-api-tls\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:38.560556 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.560524 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f6d9a2-serving-cert\"" Nov 03 07:34:38.652835 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.652790 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-metrics-certs\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:38.653034 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.652931 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e84d830-9ae7-41f5-a105-068e1d316171-openshift-service-ca\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:38.653034 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.652961 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msvlc\" (UniqueName: \"kubernetes.io/projected/7e84d830-9ae7-41f5-a105-068e1d316171-kube-api-access-msvlc\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:38.653034 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.652988 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-maas-api-tls\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:38.653201 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:38.653177 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-serving-cert: secret "maas-api-e2e-api-a-f6d9a2-serving-cert" not found Nov 03 07:34:38.653298 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:38.653273 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-maas-api-tls podName:7e84d830-9ae7-41f5-a105-068e1d316171 nodeName:}" failed. No retries permitted until 2026-11-03 07:34:39.153250973 +0000 UTC m=+2134.345792718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-maas-api-tls") pod "maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" (UID: "7e84d830-9ae7-41f5-a105-068e1d316171") : secret "maas-api-e2e-api-a-f6d9a2-serving-cert" not found Nov 03 07:34:38.653667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.653648 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e84d830-9ae7-41f5-a105-068e1d316171-openshift-service-ca\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:38.655508 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.655490 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-metrics-certs\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:38.715966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:38.715889 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msvlc\" (UniqueName: \"kubernetes.io/projected/7e84d830-9ae7-41f5-a105-068e1d316171-kube-api-access-msvlc\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:38.775997 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:38.775960 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:34:39.165971 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:39.165930 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-maas-api-tls\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:39.168614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:39.168594 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-maas-api-tls\") pod \"maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:39.376558 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:39.376526 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:39.761489 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:39.761456 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g"] Nov 03 07:34:39.763394 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:34:39.763370 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e84d830_9ae7_41f5_a105_068e1d316171.slice/crio-f51485b4222291919e4be6a14e68f334f051b131221274da2cbbe18b5f3713b9 WatchSource:0}: Error finding container f51485b4222291919e4be6a14e68f334f051b131221274da2cbbe18b5f3713b9: Status 404 returned error can't find the container with id f51485b4222291919e4be6a14e68f334f051b131221274da2cbbe18b5f3713b9 Nov 03 07:34:39.764844 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:39.764825 4409 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:34:39.879613 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:39.879575 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" podUID="769e14ab-8bb4-4734-8fb7-017a5697a095" containerName="payload-processing" containerID="cri-o://d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9" gracePeriod=25 Nov 03 07:34:39.955749 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:34:39.955683 4409 logging.go:55] [core] [Channel #3829 SubChannel #3830]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.115:50052", ServerName: "10.133.0.115:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.115:50052: connect: connection refused" Nov 03 07:34:40.001252 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.001219 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" podUID="d7c9c0b6-fff4-4390-85ee-8f244f021f6e" containerName="payload-pre-processing" containerID="cri-o://df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69" gracePeriod=25 Nov 03 07:34:40.179304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.179276 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" Nov 03 07:34:40.281265 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.281239 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" Nov 03 07:34:40.281622 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.281606 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769e14ab-8bb4-4734-8fb7-017a5697a095-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769e14ab-8bb4-4734-8fb7-017a5697a095-plugins-config-volume\") pod \"769e14ab-8bb4-4734-8fb7-017a5697a095\" (UID: \"769e14ab-8bb4-4734-8fb7-017a5697a095\") " Nov 03 07:34:40.281831 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.281815 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769e14ab-8bb4-4734-8fb7-017a5697a095-kube-api-access-k4cmm\" (UniqueName: \"kubernetes.io/projected/769e14ab-8bb4-4734-8fb7-017a5697a095-kube-api-access-k4cmm\") pod \"769e14ab-8bb4-4734-8fb7-017a5697a095\" (UID: \"769e14ab-8bb4-4734-8fb7-017a5697a095\") " Nov 03 07:34:40.281963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.281945 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769e14ab-8bb4-4734-8fb7-017a5697a095-plugins-config-volume" pod "769e14ab-8bb4-4734-8fb7-017a5697a095" (UID: "769e14ab-8bb4-4734-8fb7-017a5697a095"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:34:40.282350 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.282328 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/769e14ab-8bb4-4734-8fb7-017a5697a095-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:40.284027 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.284009 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769e14ab-8bb4-4734-8fb7-017a5697a095-kube-api-access-k4cmm" pod "769e14ab-8bb4-4734-8fb7-017a5697a095" (UID: "769e14ab-8bb4-4734-8fb7-017a5697a095"). InnerVolumeSpecName "kube-api-access-k4cmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:40.382946 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.382911 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-plugins-config-volume\") pod \"d7c9c0b6-fff4-4390-85ee-8f244f021f6e\" (UID: \"d7c9c0b6-fff4-4390-85ee-8f244f021f6e\") " Nov 03 07:34:40.383164 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.383147 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-kube-api-access-m4hsw\" (UniqueName: \"kubernetes.io/projected/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-kube-api-access-m4hsw\") pod \"d7c9c0b6-fff4-4390-85ee-8f244f021f6e\" (UID: \"d7c9c0b6-fff4-4390-85ee-8f244f021f6e\") " Nov 03 07:34:40.383354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.383320 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-plugins-config-volume" pod "d7c9c0b6-fff4-4390-85ee-8f244f021f6e" (UID: "d7c9c0b6-fff4-4390-85ee-8f244f021f6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:34:40.383772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.383754 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:40.383837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.383778 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4cmm\" (UniqueName: \"kubernetes.io/projected/769e14ab-8bb4-4734-8fb7-017a5697a095-kube-api-access-k4cmm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:40.385373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.385353 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-kube-api-access-m4hsw" pod "d7c9c0b6-fff4-4390-85ee-8f244f021f6e" (UID: "d7c9c0b6-fff4-4390-85ee-8f244f021f6e"). InnerVolumeSpecName "kube-api-access-m4hsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:40.474820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.474780 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" event={"ID":"7e84d830-9ae7-41f5-a105-068e1d316171","Type":"ContainerStarted","Data":"f51485b4222291919e4be6a14e68f334f051b131221274da2cbbe18b5f3713b9"} Nov 03 07:34:40.476178 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.476152 4409 generic.go:356] "Generic (PLEG): container finished" podID="d7c9c0b6-fff4-4390-85ee-8f244f021f6e" containerID="df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69" exitCode=0 Nov 03 07:34:40.476301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.476199 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" Nov 03 07:34:40.476301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.476236 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" event={"ID":"d7c9c0b6-fff4-4390-85ee-8f244f021f6e","Type":"ContainerDied","Data":"df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69"} Nov 03 07:34:40.476301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.476285 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25" event={"ID":"d7c9c0b6-fff4-4390-85ee-8f244f021f6e","Type":"ContainerDied","Data":"cbffde7ca8f0d4922d3d7fff572d7075056d2644abd751d6b36bc4e0b12808d0"} Nov 03 07:34:40.476471 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.476315 4409 scope.go:122] "RemoveContainer" containerID="df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69" Nov 03 07:34:40.477766 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.477745 4409 generic.go:356] "Generic (PLEG): container finished" podID="769e14ab-8bb4-4734-8fb7-017a5697a095" containerID="d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9" exitCode=0 Nov 03 07:34:40.477876 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.477777 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" Nov 03 07:34:40.477876 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.477834 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" event={"ID":"769e14ab-8bb4-4734-8fb7-017a5697a095","Type":"ContainerDied","Data":"d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9"} Nov 03 07:34:40.478003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.477889 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" event={"ID":"769e14ab-8bb4-4734-8fb7-017a5697a095","Type":"ContainerDied","Data":"647e8c9c08bfcc4fce74671f800efe585a8d905ff40fdf36d3de43e09d264083"} Nov 03 07:34:40.484980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.484950 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4hsw\" (UniqueName: \"kubernetes.io/projected/d7c9c0b6-fff4-4390-85ee-8f244f021f6e-kube-api-access-m4hsw\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:40.488325 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.488308 4409 scope.go:122] "RemoveContainer" containerID="df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69" Nov 03 07:34:40.488597 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:40.488581 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69\": container with ID starting with df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69 not found: ID does not exist" containerID="df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69" Nov 03 07:34:40.488637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.488605 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69"} err="failed to get container status \"df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69\": rpc error: code = NotFound desc = could not find container \"df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69\": container with ID starting with df36cb78955211046c579802f5c0ac0102040485702ce853bb8de2ba26cf7e69 not found: ID does not exist" Nov 03 07:34:40.488637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.488620 4409 scope.go:122] "RemoveContainer" containerID="d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9" Nov 03 07:34:40.499027 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.499006 4409 scope.go:122] "RemoveContainer" containerID="d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9" Nov 03 07:34:40.499369 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:40.499348 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9\": container with ID starting with d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9 not found: ID does not exist" containerID="d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9" Nov 03 07:34:40.499415 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.499378 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9"} err="failed to get container status \"d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9\": rpc error: code = NotFound desc = could not find container \"d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9\": container with ID starting with d4ebec6626bac65484a21d397864fce095cb7a52062bb8f64d777edad4c63ca9 not found: ID does not exist" Nov 03 07:34:40.646312 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.646282 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5"] Nov 03 07:34:40.651548 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.651521 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5"] Nov 03 07:34:40.762441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.762405 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25"] Nov 03 07:34:40.769239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.769208 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-09fd61-697687976c-75z25"] Nov 03 07:34:40.915569 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.915488 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769e14ab-8bb4-4734-8fb7-017a5697a095" path="/var/lib/kubelet/pods/769e14ab-8bb4-4734-8fb7-017a5697a095/volumes" Nov 03 07:34:40.916086 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.916068 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c9c0b6-fff4-4390-85ee-8f244f021f6e" path="/var/lib/kubelet/pods/d7c9c0b6-fff4-4390-85ee-8f244f021f6e/volumes" Nov 03 07:34:40.955660 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:40.955622 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-09fd61-5ccdf769b5-q6jl5" podUID="769e14ab-8bb4-4734-8fb7-017a5697a095" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.115:50052\" within 1s: context deadline exceeded" Nov 03 07:34:41.485904 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:41.485861 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" event={"ID":"7e84d830-9ae7-41f5-a105-068e1d316171","Type":"ContainerStarted","Data":"d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e"} Nov 03 07:34:41.486138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:41.486119 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:41.607781 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:41.607703 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" podStartSLOduration=2.7408851050000003 podStartE2EDuration="3.607687655s" totalImagesPullingTime="866.80255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:34:41.604199217 +0000 UTC m=+2136.796740991" watchObservedRunningTime="2026-11-03 07:34:41.607687655 +0000 UTC m=+2136.800229405" Nov 03 07:34:41.906748 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:41.891986 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st"] Nov 03 07:34:41.910961 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:41.910512 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerName="istio-proxy" containerID="cri-o://09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7" gracePeriod=30 Nov 03 07:34:44.978236 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:44.978204 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s"] Nov 03 07:34:44.978968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:44.978952 4409 cpu_manager.go:414] "containerMap: removing container" podUID="769e14ab-8bb4-4734-8fb7-017a5697a095" containerName="payload-processing" Nov 03 07:34:44.978968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:44.978969 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769e14ab-8bb4-4734-8fb7-017a5697a095" containerName="payload-processing" Nov 03 07:34:44.979089 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:44.978977 4409 cpu_manager.go:414] "containerMap: removing container" podUID="d7c9c0b6-fff4-4390-85ee-8f244f021f6e" containerName="payload-pre-processing" Nov 03 07:34:44.979089 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:44.978982 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c9c0b6-fff4-4390-85ee-8f244f021f6e" containerName="payload-pre-processing" Nov 03 07:34:44.979154 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:44.979090 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="769e14ab-8bb4-4734-8fb7-017a5697a095" containerName="payload-processing" Nov 03 07:34:44.979154 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:44.979099 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c9c0b6-fff4-4390-85ee-8f244f021f6e" containerName="payload-pre-processing" Nov 03 07:34:44.983971 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:44.983954 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.042809 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.042781 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0e8bf4-openshift-default-dockercfg-77x9h\"" Nov 03 07:34:45.047932 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.047906 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s"] Nov 03 07:34:45.125317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.125270 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d816bfac-359f-463e-81b9-accd32ce79e1-istio-podinfo\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.125731 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.125685 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6c4\" (UniqueName: \"kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-kube-api-access-pg6c4\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.126161 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.126117 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-certs\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.126339 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.126320 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-envoy\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.126605 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.126585 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-istio-token\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.126788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.126772 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-data\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.132435 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.132404 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-socket\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.132797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.132767 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-credential-socket\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.132981 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.132965 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d816bfac-359f-463e-81b9-accd32ce79e1-istiod-ca-cert\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.234802 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.234656 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6c4\" (UniqueName: \"kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-kube-api-access-pg6c4\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.234802 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.234742 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-certs\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.236494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.235117 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-envoy\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.236494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.235283 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-istio-token\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.236494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.235338 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-data\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.236494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.235415 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-socket\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.236494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.235497 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-credential-socket\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.236494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.235543 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d816bfac-359f-463e-81b9-accd32ce79e1-istiod-ca-cert\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.236494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.235278 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-certs\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.236494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.235979 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-socket\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.236494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.236004 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-data\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.236494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.236233 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-credential-socket\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.236494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.235595 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d816bfac-359f-463e-81b9-accd32ce79e1-istio-podinfo\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.237273 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.236691 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d816bfac-359f-463e-81b9-accd32ce79e1-istiod-ca-cert\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.254211 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.242366 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d816bfac-359f-463e-81b9-accd32ce79e1-istio-podinfo\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.254211 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.254000 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-envoy\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.300773 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.299950 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-istio-token\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.300773 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.299985 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6c4\" (UniqueName: \"kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-kube-api-access-pg6c4\") pod \"e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:45.600411 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:45.600381 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:46.037075 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:46.037049 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s"] Nov 03 07:34:46.038524 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:34:46.038501 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd816bfac_359f_463e_81b9_accd32ce79e1.slice/crio-f2b97ec0c481f956e3899a673ded9983fc7b1b864a502e9c6b2489c19e9c508f WatchSource:0}: Error finding container f2b97ec0c481f956e3899a673ded9983fc7b1b864a502e9c6b2489c19e9c508f: Status 404 returned error can't find the container with id f2b97ec0c481f956e3899a673ded9983fc7b1b864a502e9c6b2489c19e9c508f Nov 03 07:34:46.041247 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:46.041214 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:34:46.041343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:46.041280 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:34:46.041343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:46.041322 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:34:46.527095 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:46.527058 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" event={"ID":"d816bfac-359f-463e-81b9-accd32ce79e1","Type":"ContainerStarted","Data":"bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11"} Nov 03 07:34:46.527268 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:46.527111 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" event={"ID":"d816bfac-359f-463e-81b9-accd32ce79e1","Type":"ContainerStarted","Data":"f2b97ec0c481f956e3899a673ded9983fc7b1b864a502e9c6b2489c19e9c508f"} Nov 03 07:34:46.600526 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:46.600482 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:46.663586 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:46.663522 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" podStartSLOduration=2.663504364 podStartE2EDuration="2.663504364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:34:46.659003439 +0000 UTC m=+2141.851545189" watchObservedRunningTime="2026-11-03 07:34:46.663504364 +0000 UTC m=+2141.856046115" Nov 03 07:34:47.361220 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.361194 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:34:47.471058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471029 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-data\") pod \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " Nov 03 07:34:47.471227 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471081 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istiod-ca-cert\") pod \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " Nov 03 07:34:47.471227 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471111 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-socket\") pod \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " Nov 03 07:34:47.471227 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471150 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-podinfo\") pod \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " Nov 03 07:34:47.471227 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471179 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-kube-api-access-lv4vp\" (UniqueName: \"kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-kube-api-access-lv4vp\") pod \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " Nov 03 07:34:47.471465 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471269 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-envoy\") pod \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " Nov 03 07:34:47.471465 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471316 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-data" pod "d5c819ba-88c2-4593-95fe-7ba9f7fff015" (UID: "d5c819ba-88c2-4593-95fe-7ba9f7fff015"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:47.471465 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471323 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-token\" (UniqueName: \"kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-token\") pod \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " Nov 03 07:34:47.471465 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471416 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-socket" pod "d5c819ba-88c2-4593-95fe-7ba9f7fff015" (UID: "d5c819ba-88c2-4593-95fe-7ba9f7fff015"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:47.471680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471555 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-credential-socket\") pod \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " Nov 03 07:34:47.471680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471664 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-certs\") pod \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\" (UID: \"d5c819ba-88c2-4593-95fe-7ba9f7fff015\") " Nov 03 07:34:47.471837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471809 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istiod-ca-cert" pod "d5c819ba-88c2-4593-95fe-7ba9f7fff015" (UID: "d5c819ba-88c2-4593-95fe-7ba9f7fff015"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:34:47.471929 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.471903 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-credential-socket" pod "d5c819ba-88c2-4593-95fe-7ba9f7fff015" (UID: "d5c819ba-88c2-4593-95fe-7ba9f7fff015"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:47.472297 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.472211 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:47.472407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.472338 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:47.472407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.472356 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:47.472407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.472109 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-certs" pod "d5c819ba-88c2-4593-95fe-7ba9f7fff015" (UID: "d5c819ba-88c2-4593-95fe-7ba9f7fff015"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:47.473663 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.473638 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-podinfo" pod "d5c819ba-88c2-4593-95fe-7ba9f7fff015" (UID: "d5c819ba-88c2-4593-95fe-7ba9f7fff015"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:34:47.473663 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.473644 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-token" pod "d5c819ba-88c2-4593-95fe-7ba9f7fff015" (UID: "d5c819ba-88c2-4593-95fe-7ba9f7fff015"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:47.474114 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.474092 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-envoy" pod "d5c819ba-88c2-4593-95fe-7ba9f7fff015" (UID: "d5c819ba-88c2-4593-95fe-7ba9f7fff015"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:34:47.474114 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.474101 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-kube-api-access-lv4vp" pod "d5c819ba-88c2-4593-95fe-7ba9f7fff015" (UID: "d5c819ba-88c2-4593-95fe-7ba9f7fff015"). InnerVolumeSpecName "kube-api-access-lv4vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:34:47.501429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.501409 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:34:47.540114 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.540028 4409 generic.go:356] "Generic (PLEG): container finished" podID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerID="09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7" exitCode=0 Nov 03 07:34:47.540114 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.540085 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" Nov 03 07:34:47.540321 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.540123 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" event={"ID":"d5c819ba-88c2-4593-95fe-7ba9f7fff015","Type":"ContainerDied","Data":"09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7"} Nov 03 07:34:47.540321 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.540190 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st" event={"ID":"d5c819ba-88c2-4593-95fe-7ba9f7fff015","Type":"ContainerDied","Data":"f8e170475cc9aee8df9d12982e20438e733834660660c885d6caad22a9cf6061"} Nov 03 07:34:47.540321 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.540224 4409 scope.go:122] "RemoveContainer" containerID="09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7" Nov 03 07:34:47.554254 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.554234 4409 scope.go:122] "RemoveContainer" containerID="8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352" Nov 03 07:34:47.564556 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.564533 4409 scope.go:122] "RemoveContainer" containerID="09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7" Nov 03 07:34:47.564837 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:47.564819 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7\": container with ID starting with 09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7 not found: ID does not exist" containerID="09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7" Nov 03 07:34:47.564925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.564850 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7"} err="failed to get container status \"09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7\": rpc error: code = NotFound desc = could not find container \"09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7\": container with ID starting with 09aab7305ea46a1732459168f4bc75b3c74ca783ed7d1c0ef69067f25c1ba3d7 not found: ID does not exist" Nov 03 07:34:47.564925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.564876 4409 scope.go:122] "RemoveContainer" containerID="8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352" Nov 03 07:34:47.565154 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:34:47.565131 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352\": container with ID starting with 8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352 not found: ID does not exist" containerID="8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352" Nov 03 07:34:47.565198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.565164 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352"} err="failed to get container status \"8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352\": rpc error: code = NotFound desc = could not find container \"8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352\": container with ID starting with 8620c56892ad064b9f8423fc142d39987cb15a832fc8c7ca4cadf5db6b775352 not found: ID does not exist" Nov 03 07:34:47.573914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.573852 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:47.573914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.573882 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv4vp\" (UniqueName: \"kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-kube-api-access-lv4vp\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:47.573914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.573900 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:47.573914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.573915 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c819ba-88c2-4593-95fe-7ba9f7fff015-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:47.574177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.573931 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:47.574177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.573946 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c819ba-88c2-4593-95fe-7ba9f7fff015-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:34:47.607224 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.607203 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:47.864243 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.864211 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st"] Nov 03 07:34:47.870587 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:47.870563 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-418f41-openshift-default-6b76655469-5f4st"] Nov 03 07:34:48.548683 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:48.548653 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:48.549531 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:48.549516 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:34:48.914340 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:48.914307 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" path="/var/lib/kubelet/pods/d5c819ba-88c2-4593-95fe-7ba9f7fff015/volumes" Nov 03 07:34:53.293646 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.293604 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7"] Nov 03 07:34:53.294514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.294490 4409 cpu_manager.go:414] "containerMap: removing container" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerName="istio-proxy" Nov 03 07:34:53.294628 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.294514 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerName="istio-proxy" Nov 03 07:34:53.294628 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.294538 4409 cpu_manager.go:414] "containerMap: removing container" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerName="istio-proxy" Nov 03 07:34:53.294628 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.294545 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerName="istio-proxy" Nov 03 07:34:53.294805 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.294731 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerName="istio-proxy" Nov 03 07:34:53.300120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.300101 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" Nov 03 07:34:53.356976 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.356952 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7"] Nov 03 07:34:53.408789 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.408742 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg47p\" (UniqueName: \"kubernetes.io/projected/34261966-3f63-47b2-9bff-e687beb22e3e-kube-api-access-xg47p\") pod \"payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7\" (UID: \"34261966-3f63-47b2-9bff-e687beb22e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" Nov 03 07:34:53.439737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.426910 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34261966-3f63-47b2-9bff-e687beb22e3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7\" (UID: \"34261966-3f63-47b2-9bff-e687beb22e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" Nov 03 07:34:53.439737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.422103 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0e8bf4-dockercfg-w67f2\"" Nov 03 07:34:53.439737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.421972 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0e8bf4\"" Nov 03 07:34:53.531763 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.530281 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34261966-3f63-47b2-9bff-e687beb22e3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7\" (UID: \"34261966-3f63-47b2-9bff-e687beb22e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" Nov 03 07:34:53.531763 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.530348 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34261966-3f63-47b2-9bff-e687beb22e3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7\" (UID: \"34261966-3f63-47b2-9bff-e687beb22e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" Nov 03 07:34:53.531763 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.530630 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg47p\" (UniqueName: \"kubernetes.io/projected/34261966-3f63-47b2-9bff-e687beb22e3e-kube-api-access-xg47p\") pod \"payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7\" (UID: \"34261966-3f63-47b2-9bff-e687beb22e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" Nov 03 07:34:53.541397 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.541367 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9"] Nov 03 07:34:53.543051 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.543024 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c819ba-88c2-4593-95fe-7ba9f7fff015" containerName="istio-proxy" Nov 03 07:34:53.590977 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.590947 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9"] Nov 03 07:34:53.591323 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.591301 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" Nov 03 07:34:53.596957 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.596934 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg47p\" (UniqueName: \"kubernetes.io/projected/34261966-3f63-47b2-9bff-e687beb22e3e-kube-api-access-xg47p\") pod \"payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7\" (UID: \"34261966-3f63-47b2-9bff-e687beb22e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" Nov 03 07:34:53.613888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.613865 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" Nov 03 07:34:53.631533 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.631504 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff76af4-5e55-4cd1-8341-c4d674fad0fa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9\" (UID: \"dff76af4-5e55-4cd1-8341-c4d674fad0fa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" Nov 03 07:34:53.631811 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.631789 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwsk\" (UniqueName: \"kubernetes.io/projected/dff76af4-5e55-4cd1-8341-c4d674fad0fa-kube-api-access-vzwsk\") pod \"payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9\" (UID: \"dff76af4-5e55-4cd1-8341-c4d674fad0fa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" Nov 03 07:34:53.738762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.733457 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwsk\" (UniqueName: \"kubernetes.io/projected/dff76af4-5e55-4cd1-8341-c4d674fad0fa-kube-api-access-vzwsk\") pod \"payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9\" (UID: \"dff76af4-5e55-4cd1-8341-c4d674fad0fa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" Nov 03 07:34:53.738762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.733592 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff76af4-5e55-4cd1-8341-c4d674fad0fa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9\" (UID: \"dff76af4-5e55-4cd1-8341-c4d674fad0fa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" Nov 03 07:34:53.738762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.734667 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff76af4-5e55-4cd1-8341-c4d674fad0fa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9\" (UID: \"dff76af4-5e55-4cd1-8341-c4d674fad0fa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" Nov 03 07:34:53.796131 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.796103 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwsk\" (UniqueName: \"kubernetes.io/projected/dff76af4-5e55-4cd1-8341-c4d674fad0fa-kube-api-access-vzwsk\") pod \"payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9\" (UID: \"dff76af4-5e55-4cd1-8341-c4d674fad0fa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" Nov 03 07:34:53.940893 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:53.940821 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" Nov 03 07:34:54.027091 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:34:54.026940 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34261966_3f63_47b2_9bff_e687beb22e3e.slice/crio-ddd188d62813bfa15423dc8fdb09d66a2e0e2c0ef67f8ac746c0bb71ba6be4dd WatchSource:0}: Error finding container ddd188d62813bfa15423dc8fdb09d66a2e0e2c0ef67f8ac746c0bb71ba6be4dd: Status 404 returned error can't find the container with id ddd188d62813bfa15423dc8fdb09d66a2e0e2c0ef67f8ac746c0bb71ba6be4dd Nov 03 07:34:54.028389 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:54.028358 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7"] Nov 03 07:34:54.327666 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:54.327638 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9"] Nov 03 07:34:54.328965 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:34:54.328934 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff76af4_5e55_4cd1_8341_c4d674fad0fa.slice/crio-011777a73f119d95fc9afca9e58dab7c2e2903e663744b269bf4e4638d081b6e WatchSource:0}: Error finding container 011777a73f119d95fc9afca9e58dab7c2e2903e663744b269bf4e4638d081b6e: Status 404 returned error can't find the container with id 011777a73f119d95fc9afca9e58dab7c2e2903e663744b269bf4e4638d081b6e Nov 03 07:34:54.606391 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:54.606293 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" event={"ID":"34261966-3f63-47b2-9bff-e687beb22e3e","Type":"ContainerStarted","Data":"3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02"} Nov 03 07:34:54.606391 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:54.606347 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" event={"ID":"34261966-3f63-47b2-9bff-e687beb22e3e","Type":"ContainerStarted","Data":"ddd188d62813bfa15423dc8fdb09d66a2e0e2c0ef67f8ac746c0bb71ba6be4dd"} Nov 03 07:34:54.606391 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:54.606379 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" Nov 03 07:34:54.608232 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:54.608203 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" event={"ID":"dff76af4-5e55-4cd1-8341-c4d674fad0fa","Type":"ContainerStarted","Data":"38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2"} Nov 03 07:34:54.608362 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:54.608255 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" event={"ID":"dff76af4-5e55-4cd1-8341-c4d674fad0fa","Type":"ContainerStarted","Data":"011777a73f119d95fc9afca9e58dab7c2e2903e663744b269bf4e4638d081b6e"} Nov 03 07:34:54.608362 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:54.608285 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" Nov 03 07:34:54.735943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:54.735852 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" podStartSLOduration=1.735838904 podStartE2EDuration="1.735838904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:34:54.735538063 +0000 UTC m=+2149.928079816" watchObservedRunningTime="2026-11-03 07:34:54.735838904 +0000 UTC m=+2149.928380655" Nov 03 07:34:54.874447 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:34:54.874326 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" podStartSLOduration=1.874299008 podStartE2EDuration="1.874299008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:34:54.864536682 +0000 UTC m=+2150.057078443" watchObservedRunningTime="2026-11-03 07:34:54.874299008 +0000 UTC m=+2150.066840759" Nov 03 07:35:04.871102 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:04.871067 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr"] Nov 03 07:35:04.871504 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:04.871246 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" podUID="533f3d9f-f1f3-47b1-8bfb-a145524ddd4f" containerName="maas-api" containerID="cri-o://bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178" gracePeriod=30 Nov 03 07:35:05.193207 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.193183 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:35:05.236309 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.236284 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-kube-api-access-nltbm\" (UniqueName: \"kubernetes.io/projected/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-kube-api-access-nltbm\") pod \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " Nov 03 07:35:05.236451 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.236435 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-metrics-certs\") pod \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " Nov 03 07:35:05.236510 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.236494 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-openshift-service-ca\") pod \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " Nov 03 07:35:05.236575 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.236560 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-maas-api-tls\") pod \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\" (UID: \"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f\") " Nov 03 07:35:05.236873 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.236848 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-openshift-service-ca" pod "533f3d9f-f1f3-47b1-8bfb-a145524ddd4f" (UID: "533f3d9f-f1f3-47b1-8bfb-a145524ddd4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:35:05.237248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.237224 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:05.238747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.238677 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-metrics-certs" pod "533f3d9f-f1f3-47b1-8bfb-a145524ddd4f" (UID: "533f3d9f-f1f3-47b1-8bfb-a145524ddd4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:35:05.238747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.238687 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-kube-api-access-nltbm" pod "533f3d9f-f1f3-47b1-8bfb-a145524ddd4f" (UID: "533f3d9f-f1f3-47b1-8bfb-a145524ddd4f"). InnerVolumeSpecName "kube-api-access-nltbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:35:05.238863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.238774 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-maas-api-tls" pod "533f3d9f-f1f3-47b1-8bfb-a145524ddd4f" (UID: "533f3d9f-f1f3-47b1-8bfb-a145524ddd4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:35:05.349777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.347053 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nltbm\" (UniqueName: \"kubernetes.io/projected/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-kube-api-access-nltbm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:05.349777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.347092 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:05.349777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.347111 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:05.619137 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.619106 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" Nov 03 07:35:05.619303 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.619180 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" Nov 03 07:35:05.691158 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.691126 4409 generic.go:356] "Generic (PLEG): container finished" podID="533f3d9f-f1f3-47b1-8bfb-a145524ddd4f" containerID="bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178" exitCode=0 Nov 03 07:35:05.691315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.691170 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" Nov 03 07:35:05.691315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.691193 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" event={"ID":"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f","Type":"ContainerDied","Data":"bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178"} Nov 03 07:35:05.691315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.691233 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" event={"ID":"533f3d9f-f1f3-47b1-8bfb-a145524ddd4f","Type":"ContainerDied","Data":"ba9b37dfd702e937bd94e8b6aba0f9e875baa2a1afc373748d9a21ad3a114756"} Nov 03 07:35:05.691315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.691256 4409 scope.go:122] "RemoveContainer" containerID="bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178" Nov 03 07:35:05.705393 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.705370 4409 scope.go:122] "RemoveContainer" containerID="bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178" Nov 03 07:35:05.705908 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:35:05.705885 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178\": container with ID starting with bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178 not found: ID does not exist" containerID="bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178" Nov 03 07:35:05.706002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:05.705914 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178"} err="failed to get container status \"bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178\": rpc error: code = NotFound desc = could not find container \"bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178\": container with ID starting with bba9f27c1057581460085ce4839dbdf9c0a255f437ee300df0cc6ab226c5b178 not found: ID does not exist" Nov 03 07:35:06.027993 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:06.027908 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr"] Nov 03 07:35:06.067922 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:06.067884 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr"] Nov 03 07:35:06.915050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:06.915018 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533f3d9f-f1f3-47b1-8bfb-a145524ddd4f" path="/var/lib/kubelet/pods/533f3d9f-f1f3-47b1-8bfb-a145524ddd4f/volumes" Nov 03 07:35:08.115069 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:08.115029 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-6njpr" podUID="533f3d9f-f1f3-47b1-8bfb-a145524ddd4f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.108:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:35:08.774132 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:35:08.774097 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:35:15.463606 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:15.463576 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt"] Nov 03 07:35:15.464075 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:15.463790 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" containerID="cri-o://a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56" gracePeriod=30 Nov 03 07:35:16.376423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:16.376385 4409 patch_prober.go:29] interesting pod/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:35:16.376690 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:16.376450 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:35:18.192756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.189388 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm"] Nov 03 07:35:18.192756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.190931 4409 cpu_manager.go:414] "containerMap: removing container" podUID="533f3d9f-f1f3-47b1-8bfb-a145524ddd4f" containerName="maas-api" Nov 03 07:35:18.192756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.190955 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533f3d9f-f1f3-47b1-8bfb-a145524ddd4f" containerName="maas-api" Nov 03 07:35:18.205739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.204167 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="533f3d9f-f1f3-47b1-8bfb-a145524ddd4f" containerName="maas-api" Nov 03 07:35:18.214738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.211532 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" Nov 03 07:35:18.237735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.235304 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm"] Nov 03 07:35:18.294220 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.280095 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-295fba7c\"" Nov 03 07:35:18.294220 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.280297 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-295fba7c-dockercfg-44r4r\"" Nov 03 07:35:18.336215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.336173 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v"] Nov 03 07:35:18.338656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.338619 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkppp\" (UniqueName: \"kubernetes.io/projected/c8b8b2c6-1104-477c-af9c-5fe063f450c5-kube-api-access-rkppp\") pod \"payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm\" (UID: \"c8b8b2c6-1104-477c-af9c-5fe063f450c5\") " pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" Nov 03 07:35:18.338875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.338854 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b8b2c6-1104-477c-af9c-5fe063f450c5-plugins-config-volume\") pod \"payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm\" (UID: \"c8b8b2c6-1104-477c-af9c-5fe063f450c5\") " pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" Nov 03 07:35:18.341927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.341884 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" Nov 03 07:35:18.390795 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.390767 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v"] Nov 03 07:35:18.440519 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.440488 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b8b2c6-1104-477c-af9c-5fe063f450c5-plugins-config-volume\") pod \"payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm\" (UID: \"c8b8b2c6-1104-477c-af9c-5fe063f450c5\") " pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" Nov 03 07:35:18.440663 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.440646 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkppp\" (UniqueName: \"kubernetes.io/projected/c8b8b2c6-1104-477c-af9c-5fe063f450c5-kube-api-access-rkppp\") pod \"payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm\" (UID: \"c8b8b2c6-1104-477c-af9c-5fe063f450c5\") " pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" Nov 03 07:35:18.440829 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.440806 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5714b44e-b0be-419d-8106-94e5a28f9e55-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v\" (UID: \"5714b44e-b0be-419d-8106-94e5a28f9e55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" Nov 03 07:35:18.440940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.440921 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqzc9\" (UniqueName: \"kubernetes.io/projected/5714b44e-b0be-419d-8106-94e5a28f9e55-kube-api-access-pqzc9\") pod \"payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v\" (UID: \"5714b44e-b0be-419d-8106-94e5a28f9e55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" Nov 03 07:35:18.441267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.441250 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b8b2c6-1104-477c-af9c-5fe063f450c5-plugins-config-volume\") pod \"payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm\" (UID: \"c8b8b2c6-1104-477c-af9c-5fe063f450c5\") " pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" Nov 03 07:35:18.503778 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.503697 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkppp\" (UniqueName: \"kubernetes.io/projected/c8b8b2c6-1104-477c-af9c-5fe063f450c5-kube-api-access-rkppp\") pod \"payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm\" (UID: \"c8b8b2c6-1104-477c-af9c-5fe063f450c5\") " pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" Nov 03 07:35:18.541691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.541659 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5714b44e-b0be-419d-8106-94e5a28f9e55-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v\" (UID: \"5714b44e-b0be-419d-8106-94e5a28f9e55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" Nov 03 07:35:18.541855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.541767 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqzc9\" (UniqueName: \"kubernetes.io/projected/5714b44e-b0be-419d-8106-94e5a28f9e55-kube-api-access-pqzc9\") pod \"payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v\" (UID: \"5714b44e-b0be-419d-8106-94e5a28f9e55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" Nov 03 07:35:18.542313 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.542295 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5714b44e-b0be-419d-8106-94e5a28f9e55-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v\" (UID: \"5714b44e-b0be-419d-8106-94e5a28f9e55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" Nov 03 07:35:18.604665 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.604640 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqzc9\" (UniqueName: \"kubernetes.io/projected/5714b44e-b0be-419d-8106-94e5a28f9e55-kube-api-access-pqzc9\") pod \"payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v\" (UID: \"5714b44e-b0be-419d-8106-94e5a28f9e55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" Nov 03 07:35:18.632685 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.632664 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" Nov 03 07:35:18.659974 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:18.659950 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" Nov 03 07:35:19.020914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:19.020892 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm"] Nov 03 07:35:19.021534 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:35:19.021507 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b8b2c6_1104_477c_af9c_5fe063f450c5.slice/crio-cddd9d69b1c2c80d6c6294e8f48a7b626c02c41264bcf15581425487de72cbba WatchSource:0}: Error finding container cddd9d69b1c2c80d6c6294e8f48a7b626c02c41264bcf15581425487de72cbba: Status 404 returned error can't find the container with id cddd9d69b1c2c80d6c6294e8f48a7b626c02c41264bcf15581425487de72cbba Nov 03 07:35:19.066767 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:19.066701 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v"] Nov 03 07:35:19.067210 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:35:19.067183 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5714b44e_b0be_419d_8106_94e5a28f9e55.slice/crio-606f8c38733cadd8e3e68a94de82d9e1d276da8673988d4f2fb2bda6db3952c7 WatchSource:0}: Error finding container 606f8c38733cadd8e3e68a94de82d9e1d276da8673988d4f2fb2bda6db3952c7: Status 404 returned error can't find the container with id 606f8c38733cadd8e3e68a94de82d9e1d276da8673988d4f2fb2bda6db3952c7 Nov 03 07:35:19.800947 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:19.800904 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" event={"ID":"5714b44e-b0be-419d-8106-94e5a28f9e55","Type":"ContainerStarted","Data":"e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e"} Nov 03 07:35:19.801445 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:19.801007 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" event={"ID":"5714b44e-b0be-419d-8106-94e5a28f9e55","Type":"ContainerStarted","Data":"606f8c38733cadd8e3e68a94de82d9e1d276da8673988d4f2fb2bda6db3952c7"} Nov 03 07:35:19.801445 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:19.801045 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" Nov 03 07:35:19.802675 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:19.802652 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" event={"ID":"c8b8b2c6-1104-477c-af9c-5fe063f450c5","Type":"ContainerStarted","Data":"c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0"} Nov 03 07:35:19.802797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:19.802700 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" event={"ID":"c8b8b2c6-1104-477c-af9c-5fe063f450c5","Type":"ContainerStarted","Data":"cddd9d69b1c2c80d6c6294e8f48a7b626c02c41264bcf15581425487de72cbba"} Nov 03 07:35:19.802797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:19.802734 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" Nov 03 07:35:19.921076 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:19.921029 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" podStartSLOduration=1.921015521 podStartE2EDuration="1.921015521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:35:19.91831198 +0000 UTC m=+2175.110853730" watchObservedRunningTime="2026-11-03 07:35:19.921015521 +0000 UTC m=+2175.113557270" Nov 03 07:35:20.048083 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.048020 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" podStartSLOduration=2.047996922 podStartE2EDuration="2.047996922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:35:20.038125098 +0000 UTC m=+2175.230666849" watchObservedRunningTime="2026-11-03 07:35:20.047996922 +0000 UTC m=+2175.240538673" Nov 03 07:35:20.747705 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.747643 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:35:20.810705 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.810674 4409 generic.go:356] "Generic (PLEG): container finished" podID="49c96119-0bed-477e-9992-eb73a9761fca" containerID="a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56" exitCode=0 Nov 03 07:35:20.811122 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.810759 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" Nov 03 07:35:20.811122 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.810759 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" event={"ID":"49c96119-0bed-477e-9992-eb73a9761fca","Type":"ContainerDied","Data":"a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56"} Nov 03 07:35:20.811122 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.810825 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt" event={"ID":"49c96119-0bed-477e-9992-eb73a9761fca","Type":"ContainerDied","Data":"c7e4d1c586e50f9eef8fd3a6ebf7994179a5b8c3836a29907d3fcf5cbe9db084"} Nov 03 07:35:20.811122 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.810852 4409 scope.go:122] "RemoveContainer" containerID="a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56" Nov 03 07:35:20.821072 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.821055 4409 scope.go:122] "RemoveContainer" containerID="a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56" Nov 03 07:35:20.821341 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:35:20.821324 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56\": container with ID starting with a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56 not found: ID does not exist" containerID="a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56" Nov 03 07:35:20.821381 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.821350 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56"} err="failed to get container status \"a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56\": rpc error: code = NotFound desc = could not find container \"a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56\": container with ID starting with a5c8d3f8f03689869d034600ff51b93cbc4449a0c6252607131d7a3646527f56 not found: ID does not exist" Nov 03 07:35:20.877931 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.877906 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-istio-token\" (UniqueName: \"kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-istio-token\") pod \"49c96119-0bed-477e-9992-eb73a9761fca\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " Nov 03 07:35:20.878036 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.877949 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49c96119-0bed-477e-9992-eb73a9761fca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c96119-0bed-477e-9992-eb73a9761fca-istio-podinfo\") pod \"49c96119-0bed-477e-9992-eb73a9761fca\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " Nov 03 07:35:20.878167 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.878147 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-kube-api-access-bdswf\" (UniqueName: \"kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-kube-api-access-bdswf\") pod \"49c96119-0bed-477e-9992-eb73a9761fca\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " Nov 03 07:35:20.878259 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.878245 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-credential-socket\") pod \"49c96119-0bed-477e-9992-eb73a9761fca\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " Nov 03 07:35:20.878320 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.878296 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-socket\") pod \"49c96119-0bed-477e-9992-eb73a9761fca\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " Nov 03 07:35:20.878376 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.878350 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-certs\") pod \"49c96119-0bed-477e-9992-eb73a9761fca\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " Nov 03 07:35:20.878429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.878413 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c96119-0bed-477e-9992-eb73a9761fca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c96119-0bed-477e-9992-eb73a9761fca-istiod-ca-cert\") pod \"49c96119-0bed-477e-9992-eb73a9761fca\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " Nov 03 07:35:20.878555 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.878531 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-envoy\") pod \"49c96119-0bed-477e-9992-eb73a9761fca\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " Nov 03 07:35:20.878616 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.878565 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-credential-socket" pod "49c96119-0bed-477e-9992-eb73a9761fca" (UID: "49c96119-0bed-477e-9992-eb73a9761fca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:20.878685 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.878667 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-data\") pod \"49c96119-0bed-477e-9992-eb73a9761fca\" (UID: \"49c96119-0bed-477e-9992-eb73a9761fca\") " Nov 03 07:35:20.879078 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.879016 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-data" pod "49c96119-0bed-477e-9992-eb73a9761fca" (UID: "49c96119-0bed-477e-9992-eb73a9761fca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:20.879078 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.879031 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-socket" pod "49c96119-0bed-477e-9992-eb73a9761fca" (UID: "49c96119-0bed-477e-9992-eb73a9761fca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:20.879306 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.879208 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-certs" pod "49c96119-0bed-477e-9992-eb73a9761fca" (UID: "49c96119-0bed-477e-9992-eb73a9761fca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:20.879306 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.879138 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c96119-0bed-477e-9992-eb73a9761fca-istiod-ca-cert" pod "49c96119-0bed-477e-9992-eb73a9761fca" (UID: "49c96119-0bed-477e-9992-eb73a9761fca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:35:20.879496 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.879472 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:20.879540 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.879503 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:20.879540 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.879518 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:20.879540 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.879531 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:20.879657 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.879546 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c96119-0bed-477e-9992-eb73a9761fca-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:20.880557 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.880536 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-istio-token" pod "49c96119-0bed-477e-9992-eb73a9761fca" (UID: "49c96119-0bed-477e-9992-eb73a9761fca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:35:20.880633 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.880613 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-kube-api-access-bdswf" pod "49c96119-0bed-477e-9992-eb73a9761fca" (UID: "49c96119-0bed-477e-9992-eb73a9761fca"). InnerVolumeSpecName "kube-api-access-bdswf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:35:20.880965 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.880945 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49c96119-0bed-477e-9992-eb73a9761fca-istio-podinfo" pod "49c96119-0bed-477e-9992-eb73a9761fca" (UID: "49c96119-0bed-477e-9992-eb73a9761fca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:35:20.881124 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.881104 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-envoy" pod "49c96119-0bed-477e-9992-eb73a9761fca" (UID: "49c96119-0bed-477e-9992-eb73a9761fca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:20.981611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.981580 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:20.981772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.981614 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c96119-0bed-477e-9992-eb73a9761fca-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:20.981772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.981632 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdswf\" (UniqueName: \"kubernetes.io/projected/49c96119-0bed-477e-9992-eb73a9761fca-kube-api-access-bdswf\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:20.981772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:20.981650 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c96119-0bed-477e-9992-eb73a9761fca-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:21.230937 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:21.230910 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt"] Nov 03 07:35:21.237154 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:21.237123 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-jmmtt"] Nov 03 07:35:22.092863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.092833 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz"] Nov 03 07:35:22.093312 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.093016 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" podUID="0d7f5b7f-0ac7-40d4-be81-cd09336f25b8" containerName="maas-api" containerID="cri-o://cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9" gracePeriod=30 Nov 03 07:35:22.438694 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.438672 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:35:22.601033 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.600993 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-kube-api-access-8x4r7\" (UniqueName: \"kubernetes.io/projected/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-kube-api-access-8x4r7\") pod \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " Nov 03 07:35:22.601249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.601071 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-metrics-certs\") pod \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " Nov 03 07:35:22.601331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.601298 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-maas-api-tls\") pod \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " Nov 03 07:35:22.601427 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.601359 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-openshift-service-ca\") pod \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\" (UID: \"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8\") " Nov 03 07:35:22.602420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.602385 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-openshift-service-ca" pod "0d7f5b7f-0ac7-40d4-be81-cd09336f25b8" (UID: "0d7f5b7f-0ac7-40d4-be81-cd09336f25b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:35:22.613926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.613894 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-maas-api-tls" pod "0d7f5b7f-0ac7-40d4-be81-cd09336f25b8" (UID: "0d7f5b7f-0ac7-40d4-be81-cd09336f25b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:35:22.614048 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.613970 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-metrics-certs" pod "0d7f5b7f-0ac7-40d4-be81-cd09336f25b8" (UID: "0d7f5b7f-0ac7-40d4-be81-cd09336f25b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:35:22.614119 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.614098 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-kube-api-access-8x4r7" pod "0d7f5b7f-0ac7-40d4-be81-cd09336f25b8" (UID: "0d7f5b7f-0ac7-40d4-be81-cd09336f25b8"). InnerVolumeSpecName "kube-api-access-8x4r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:35:22.707785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.704666 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:22.707785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.704728 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:22.707785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.704745 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x4r7\" (UniqueName: \"kubernetes.io/projected/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-kube-api-access-8x4r7\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:22.707785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.704758 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:22.826949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.826913 4409 generic.go:356] "Generic (PLEG): container finished" podID="0d7f5b7f-0ac7-40d4-be81-cd09336f25b8" containerID="cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9" exitCode=0 Nov 03 07:35:22.827115 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.826972 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" Nov 03 07:35:22.827115 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.826981 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" event={"ID":"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8","Type":"ContainerDied","Data":"cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9"} Nov 03 07:35:22.827115 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.827051 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz" event={"ID":"0d7f5b7f-0ac7-40d4-be81-cd09336f25b8","Type":"ContainerDied","Data":"745cb7fe4b259ffcd7de00e6072090cafd186f0b13c6224a54d8491082b3aae0"} Nov 03 07:35:22.827115 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.827074 4409 scope.go:122] "RemoveContainer" containerID="cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9" Nov 03 07:35:22.838250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.838232 4409 scope.go:122] "RemoveContainer" containerID="cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9" Nov 03 07:35:22.838523 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:35:22.838502 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9\": container with ID starting with cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9 not found: ID does not exist" containerID="cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9" Nov 03 07:35:22.838606 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.838531 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9"} err="failed to get container status \"cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9\": rpc error: code = NotFound desc = could not find container \"cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9\": container with ID starting with cae33ac792234ce73f8df700e7dfaa44a4cc534680dd78266d333b40ee355dc9 not found: ID does not exist" Nov 03 07:35:22.914085 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.914059 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c96119-0bed-477e-9992-eb73a9761fca" path="/var/lib/kubelet/pods/49c96119-0bed-477e-9992-eb73a9761fca/volumes" Nov 03 07:35:22.995511 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:22.995438 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz"] Nov 03 07:35:23.050737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:23.049201 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5183ae-574558857f-x8twz"] Nov 03 07:35:24.914562 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:24.914529 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d7f5b7f-0ac7-40d4-be81-cd09336f25b8" path="/var/lib/kubelet/pods/0d7f5b7f-0ac7-40d4-be81-cd09336f25b8/volumes" Nov 03 07:35:28.517266 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.517226 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q"] Nov 03 07:35:28.517773 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.517428 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" podUID="01edfa30-1ac5-411a-a084-933f1a3578dd" containerName="maas-api" containerID="cri-o://033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf" gracePeriod=30 Nov 03 07:35:28.843414 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.843390 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:35:28.868658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.868633 4409 generic.go:356] "Generic (PLEG): container finished" podID="01edfa30-1ac5-411a-a084-933f1a3578dd" containerID="033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf" exitCode=0 Nov 03 07:35:28.868831 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.868680 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" Nov 03 07:35:28.868831 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.868740 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" event={"ID":"01edfa30-1ac5-411a-a084-933f1a3578dd","Type":"ContainerDied","Data":"033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf"} Nov 03 07:35:28.868831 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.868806 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q" event={"ID":"01edfa30-1ac5-411a-a084-933f1a3578dd","Type":"ContainerDied","Data":"7232451c08973376524fcb32dc07489e750d2433f33f07d758dd124f47823009"} Nov 03 07:35:28.869464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.868836 4409 scope.go:122] "RemoveContainer" containerID="033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf" Nov 03 07:35:28.878931 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.878916 4409 scope.go:122] "RemoveContainer" containerID="033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf" Nov 03 07:35:28.879209 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:35:28.879190 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf\": container with ID starting with 033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf not found: ID does not exist" containerID="033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf" Nov 03 07:35:28.879256 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.879220 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf"} err="failed to get container status \"033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf\": rpc error: code = NotFound desc = could not find container \"033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf\": container with ID starting with 033742aa66ab1c422b4cbcff6f2e44fa8ce7789bb926ec0f93ae4345e6414ccf not found: ID does not exist" Nov 03 07:35:28.990807 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.990188 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-maas-api-tls\") pod \"01edfa30-1ac5-411a-a084-933f1a3578dd\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " Nov 03 07:35:28.990807 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.990663 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01edfa30-1ac5-411a-a084-933f1a3578dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01edfa30-1ac5-411a-a084-933f1a3578dd-openshift-service-ca\") pod \"01edfa30-1ac5-411a-a084-933f1a3578dd\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " Nov 03 07:35:28.991058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.991021 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01edfa30-1ac5-411a-a084-933f1a3578dd-openshift-service-ca" pod "01edfa30-1ac5-411a-a084-933f1a3578dd" (UID: "01edfa30-1ac5-411a-a084-933f1a3578dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:35:28.991216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.991197 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01edfa30-1ac5-411a-a084-933f1a3578dd-kube-api-access-8n26k\" (UniqueName: \"kubernetes.io/projected/01edfa30-1ac5-411a-a084-933f1a3578dd-kube-api-access-8n26k\") pod \"01edfa30-1ac5-411a-a084-933f1a3578dd\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " Nov 03 07:35:28.991432 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.991409 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-metrics-certs\") pod \"01edfa30-1ac5-411a-a084-933f1a3578dd\" (UID: \"01edfa30-1ac5-411a-a084-933f1a3578dd\") " Nov 03 07:35:28.993111 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.993011 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01edfa30-1ac5-411a-a084-933f1a3578dd-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:28.999494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.998907 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-maas-api-tls" pod "01edfa30-1ac5-411a-a084-933f1a3578dd" (UID: "01edfa30-1ac5-411a-a084-933f1a3578dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:35:28.999494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:28.999311 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-metrics-certs" pod "01edfa30-1ac5-411a-a084-933f1a3578dd" (UID: "01edfa30-1ac5-411a-a084-933f1a3578dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:35:29.007772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:29.005296 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01edfa30-1ac5-411a-a084-933f1a3578dd-kube-api-access-8n26k" pod "01edfa30-1ac5-411a-a084-933f1a3578dd" (UID: "01edfa30-1ac5-411a-a084-933f1a3578dd"). InnerVolumeSpecName "kube-api-access-8n26k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:35:29.094067 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:29.093999 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:29.094067 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:29.094023 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n26k\" (UniqueName: \"kubernetes.io/projected/01edfa30-1ac5-411a-a084-933f1a3578dd-kube-api-access-8n26k\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:29.094067 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:29.094034 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01edfa30-1ac5-411a-a084-933f1a3578dd-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:29.293240 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:29.293199 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q"] Nov 03 07:35:29.299029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:29.299005 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-acdd08-d9b47b88c-7nv7q"] Nov 03 07:35:30.814231 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:30.814198 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" Nov 03 07:35:30.814638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:30.814282 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" Nov 03 07:35:30.914186 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:30.914157 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01edfa30-1ac5-411a-a084-933f1a3578dd" path="/var/lib/kubelet/pods/01edfa30-1ac5-411a-a084-933f1a3578dd/volumes" Nov 03 07:35:38.793979 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:35:38.793942 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:35:38.807136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:38.807105 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g"] Nov 03 07:35:38.807344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:38.807322 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" containerID="cri-o://31a3001db283caa1629561eb98787e75ff0908ccde5bff9c5d18dc546eba96c0" gracePeriod=30 Nov 03 07:35:44.040948 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.040915 4409 generic.go:356] "Generic (PLEG): container finished" podID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerID="31a3001db283caa1629561eb98787e75ff0908ccde5bff9c5d18dc546eba96c0" exitCode=0 Nov 03 07:35:44.041420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.040991 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" event={"ID":"5cc8eec3-9498-4244-afe5-456b9989e5f9","Type":"ContainerDied","Data":"31a3001db283caa1629561eb98787e75ff0908ccde5bff9c5d18dc546eba96c0"} Nov 03 07:35:44.041420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.041082 4409 scope.go:122] "RemoveContainer" containerID="d78af7f55956bbafcc7891881249ab41125e112f5db8fc8fed77b969a750ebe1" Nov 03 07:35:44.090979 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.090949 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:35:44.176865 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.176826 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-kube-api-access-74v2z\" (UniqueName: \"kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-kube-api-access-74v2z\") pod \"5cc8eec3-9498-4244-afe5-456b9989e5f9\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " Nov 03 07:35:44.177032 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.176899 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-token\" (UniqueName: \"kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-token\") pod \"5cc8eec3-9498-4244-afe5-456b9989e5f9\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " Nov 03 07:35:44.177032 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.176961 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-socket\") pod \"5cc8eec3-9498-4244-afe5-456b9989e5f9\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " Nov 03 07:35:44.177032 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.176996 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-envoy\") pod \"5cc8eec3-9498-4244-afe5-456b9989e5f9\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " Nov 03 07:35:44.177296 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.177079 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-credential-socket\") pod \"5cc8eec3-9498-4244-afe5-456b9989e5f9\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " Nov 03 07:35:44.177296 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.177217 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc8eec3-9498-4244-afe5-456b9989e5f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc8eec3-9498-4244-afe5-456b9989e5f9-istiod-ca-cert\") pod \"5cc8eec3-9498-4244-afe5-456b9989e5f9\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " Nov 03 07:35:44.177296 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.177273 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-socket" pod "5cc8eec3-9498-4244-afe5-456b9989e5f9" (UID: "5cc8eec3-9498-4244-afe5-456b9989e5f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:44.177468 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.177299 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-certs\") pod \"5cc8eec3-9498-4244-afe5-456b9989e5f9\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " Nov 03 07:35:44.177468 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.177346 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-credential-socket" pod "5cc8eec3-9498-4244-afe5-456b9989e5f9" (UID: "5cc8eec3-9498-4244-afe5-456b9989e5f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:44.177468 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.177364 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-data\") pod \"5cc8eec3-9498-4244-afe5-456b9989e5f9\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " Nov 03 07:35:44.177468 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.177415 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-podinfo\") pod \"5cc8eec3-9498-4244-afe5-456b9989e5f9\" (UID: \"5cc8eec3-9498-4244-afe5-456b9989e5f9\") " Nov 03 07:35:44.177667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.177529 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-certs" pod "5cc8eec3-9498-4244-afe5-456b9989e5f9" (UID: "5cc8eec3-9498-4244-afe5-456b9989e5f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:44.177667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.177623 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-data" pod "5cc8eec3-9498-4244-afe5-456b9989e5f9" (UID: "5cc8eec3-9498-4244-afe5-456b9989e5f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:44.177667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.177637 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc8eec3-9498-4244-afe5-456b9989e5f9-istiod-ca-cert" pod "5cc8eec3-9498-4244-afe5-456b9989e5f9" (UID: "5cc8eec3-9498-4244-afe5-456b9989e5f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:35:44.178335 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.178311 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:44.178425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.178340 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:44.178425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.178355 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc8eec3-9498-4244-afe5-456b9989e5f9-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:44.178425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.178369 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:44.178425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.178386 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:44.179409 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.179385 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-token" pod "5cc8eec3-9498-4244-afe5-456b9989e5f9" (UID: "5cc8eec3-9498-4244-afe5-456b9989e5f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:35:44.179514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.179496 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-envoy" pod "5cc8eec3-9498-4244-afe5-456b9989e5f9" (UID: "5cc8eec3-9498-4244-afe5-456b9989e5f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:44.179611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.179594 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-kube-api-access-74v2z" pod "5cc8eec3-9498-4244-afe5-456b9989e5f9" (UID: "5cc8eec3-9498-4244-afe5-456b9989e5f9"). InnerVolumeSpecName "kube-api-access-74v2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:35:44.179673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.179659 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-podinfo" pod "5cc8eec3-9498-4244-afe5-456b9989e5f9" (UID: "5cc8eec3-9498-4244-afe5-456b9989e5f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:35:44.279636 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.279599 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:44.279636 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.279625 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74v2z\" (UniqueName: \"kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-kube-api-access-74v2z\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:44.279636 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.279637 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:44.279636 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:44.279646 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc8eec3-9498-4244-afe5-456b9989e5f9-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:45.049154 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:45.049120 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" Nov 03 07:35:45.049154 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:45.049145 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g" event={"ID":"5cc8eec3-9498-4244-afe5-456b9989e5f9","Type":"ContainerDied","Data":"9239bb4d6b44865705091e88f455811b0d9499eec62919c318b721446870b94f"} Nov 03 07:35:45.049690 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:45.049209 4409 scope.go:122] "RemoveContainer" containerID="31a3001db283caa1629561eb98787e75ff0908ccde5bff9c5d18dc546eba96c0" Nov 03 07:35:45.291081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:45.291002 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g"] Nov 03 07:35:45.340864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:45.340830 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-acdd08-openshift-default-8686dcb677-lnf9g"] Nov 03 07:35:46.357145 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:46.357104 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv"] Nov 03 07:35:46.914442 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:46.914410 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" path="/var/lib/kubelet/pods/5cc8eec3-9498-4244-afe5-456b9989e5f9/volumes" Nov 03 07:35:50.693002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:50.692970 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn"] Nov 03 07:35:50.693514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:50.693194 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerName="istio-proxy" containerID="cri-o://80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475" gracePeriod=30 Nov 03 07:35:51.358256 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:51.358213 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" podUID="1d20a97a-9ef7-4e1a-b97e-5d420f609b59" containerName="payload-pre-processing" containerID="cri-o://0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff" gracePeriod=25 Nov 03 07:35:51.700839 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:51.700794 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" Nov 03 07:35:51.827293 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:51.827260 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-kube-api-access-pbv4w\" (UniqueName: \"kubernetes.io/projected/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-kube-api-access-pbv4w\") pod \"1d20a97a-9ef7-4e1a-b97e-5d420f609b59\" (UID: \"1d20a97a-9ef7-4e1a-b97e-5d420f609b59\") " Nov 03 07:35:51.827542 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:51.827387 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-plugins-config-volume\") pod \"1d20a97a-9ef7-4e1a-b97e-5d420f609b59\" (UID: \"1d20a97a-9ef7-4e1a-b97e-5d420f609b59\") " Nov 03 07:35:51.864254 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:51.854907 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-kube-api-access-pbv4w" pod "1d20a97a-9ef7-4e1a-b97e-5d420f609b59" (UID: "1d20a97a-9ef7-4e1a-b97e-5d420f609b59"). InnerVolumeSpecName "kube-api-access-pbv4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:35:51.864254 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:51.854965 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-plugins-config-volume" pod "1d20a97a-9ef7-4e1a-b97e-5d420f609b59" (UID: "1d20a97a-9ef7-4e1a-b97e-5d420f609b59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:35:51.932328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:51.932247 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:51.932328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:51.932279 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbv4w\" (UniqueName: \"kubernetes.io/projected/1d20a97a-9ef7-4e1a-b97e-5d420f609b59-kube-api-access-pbv4w\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:52.098522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:52.098491 4409 generic.go:356] "Generic (PLEG): container finished" podID="1d20a97a-9ef7-4e1a-b97e-5d420f609b59" containerID="0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff" exitCode=0 Nov 03 07:35:52.098693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:52.098540 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" Nov 03 07:35:52.098693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:52.098567 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" event={"ID":"1d20a97a-9ef7-4e1a-b97e-5d420f609b59","Type":"ContainerDied","Data":"0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff"} Nov 03 07:35:52.098693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:52.098616 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv" event={"ID":"1d20a97a-9ef7-4e1a-b97e-5d420f609b59","Type":"ContainerDied","Data":"c0eb20193ffc6d2e4f39eefea5d1bff31b77f94f09adc74c457e7c18626a6190"} Nov 03 07:35:52.098693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:52.098638 4409 scope.go:122] "RemoveContainer" containerID="0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff" Nov 03 07:35:52.109265 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:52.109247 4409 scope.go:122] "RemoveContainer" containerID="0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff" Nov 03 07:35:52.109550 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:35:52.109531 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff\": container with ID starting with 0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff not found: ID does not exist" containerID="0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff" Nov 03 07:35:52.109599 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:52.109557 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff"} err="failed to get container status \"0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff\": rpc error: code = NotFound desc = could not find container \"0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff\": container with ID starting with 0085fdf379f18481656a307855268cfec6faf2b02275ae68578a238c945aaaff not found: ID does not exist" Nov 03 07:35:52.266957 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:52.266891 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv"] Nov 03 07:35:52.272834 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:52.272810 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-32b500-645dbb78bc-8tmpv"] Nov 03 07:35:52.919981 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:52.919948 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d20a97a-9ef7-4e1a-b97e-5d420f609b59" path="/var/lib/kubelet/pods/1d20a97a-9ef7-4e1a-b97e-5d420f609b59/volumes" Nov 03 07:35:55.959395 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:55.959343 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:35:56.115047 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115016 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-credential-socket\") pod \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " Nov 03 07:35:56.115185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115064 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-podinfo\") pod \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " Nov 03 07:35:56.115185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115111 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istiod-ca-cert\") pod \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " Nov 03 07:35:56.115185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115138 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-envoy\") pod \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " Nov 03 07:35:56.115355 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115206 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-data\") pod \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " Nov 03 07:35:56.115355 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115263 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-token\" (UniqueName: \"kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-token\") pod \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " Nov 03 07:35:56.115355 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115296 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-credential-socket" pod "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" (UID: "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:56.115355 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115324 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-certs\") pod \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " Nov 03 07:35:56.115572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115372 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-kube-api-access-bz44k\" (UniqueName: \"kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-kube-api-access-bz44k\") pod \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " Nov 03 07:35:56.115572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115428 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-socket\") pod \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\" (UID: \"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91\") " Nov 03 07:35:56.115676 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115543 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-data" pod "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" (UID: "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:56.115676 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115619 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-certs" pod "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" (UID: "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:56.115676 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115551 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istiod-ca-cert" pod "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" (UID: "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:35:56.115905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.115863 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-socket" pod "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" (UID: "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:56.116233 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.116211 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:56.116318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.116239 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:56.116318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.116254 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:56.116318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.116268 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:56.116318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.116282 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:56.117600 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.117574 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-podinfo" pod "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" (UID: "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:35:56.117724 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.117615 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-envoy" pod "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" (UID: "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:35:56.117874 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.117856 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-token" pod "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" (UID: "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:35:56.118044 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.118028 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-kube-api-access-bz44k" pod "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" (UID: "7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91"). InnerVolumeSpecName "kube-api-access-bz44k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:35:56.161637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.161601 4409 generic.go:356] "Generic (PLEG): container finished" podID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerID="80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475" exitCode=0 Nov 03 07:35:56.161801 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.161768 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" event={"ID":"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91","Type":"ContainerDied","Data":"80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475"} Nov 03 07:35:56.161883 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.161870 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" event={"ID":"7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91","Type":"ContainerDied","Data":"c4c144ac4e95d2fe8b85952a7a669eac26b328140a3c42f4b3cf56bca8603ebf"} Nov 03 07:35:56.161941 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.161911 4409 scope.go:122] "RemoveContainer" containerID="80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475" Nov 03 07:35:56.162124 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.162109 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn" Nov 03 07:35:56.179729 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.179690 4409 scope.go:122] "RemoveContainer" containerID="762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a" Nov 03 07:35:56.189787 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.189764 4409 scope.go:122] "RemoveContainer" containerID="80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475" Nov 03 07:35:56.190070 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:35:56.190051 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475\": container with ID starting with 80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475 not found: ID does not exist" containerID="80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475" Nov 03 07:35:56.190142 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.190077 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475"} err="failed to get container status \"80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475\": rpc error: code = NotFound desc = could not find container \"80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475\": container with ID starting with 80f009d3c98a7ccfe9364c3f25a4a04ebb82341a5a7d90e0f20d62e6aa877475 not found: ID does not exist" Nov 03 07:35:56.190142 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.190094 4409 scope.go:122] "RemoveContainer" containerID="762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a" Nov 03 07:35:56.190335 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:35:56.190318 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a\": container with ID starting with 762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a not found: ID does not exist" containerID="762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a" Nov 03 07:35:56.190389 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.190339 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a"} err="failed to get container status \"762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a\": rpc error: code = NotFound desc = could not find container \"762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a\": container with ID starting with 762db0bc26a2d60ca9c1378a312e185aa8e4511c981faf8d7a19fc8d0a547a4a not found: ID does not exist" Nov 03 07:35:56.217454 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.217409 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:56.217454 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.217433 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz44k\" (UniqueName: \"kubernetes.io/projected/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-kube-api-access-bz44k\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:56.217454 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.217443 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:56.217454 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.217452 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:35:56.331935 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.331899 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn"] Nov 03 07:35:56.339274 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.339250 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-09fd61-openshift-default-6dcb688b87-9sztn"] Nov 03 07:35:56.916371 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:35:56.916331 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" path="/var/lib/kubelet/pods/7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91/volumes" Nov 03 07:36:03.632074 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:03.632042 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8"] Nov 03 07:36:03.722151 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:03.722118 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk"] Nov 03 07:36:08.633067 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:08.633010 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" podUID="a12866e2-730f-4719-b736-a337032baa17" containerName="payload-processing" containerID="cri-o://163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461" gracePeriod=25 Nov 03 07:36:08.723194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:08.723142 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" podUID="c96afb9c-3e74-479f-b1d2-f8dfce47b70e" containerName="payload-pre-processing" containerID="cri-o://a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d" gracePeriod=25 Nov 03 07:36:08.781132 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:36:08.781092 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:36:09.006325 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.006298 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" Nov 03 07:36:09.053092 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.053070 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" Nov 03 07:36:09.094668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.094644 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-kube-api-access-v6hcb\" (UniqueName: \"kubernetes.io/projected/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-kube-api-access-v6hcb\") pod \"c96afb9c-3e74-479f-b1d2-f8dfce47b70e\" (UID: \"c96afb9c-3e74-479f-b1d2-f8dfce47b70e\") " Nov 03 07:36:09.094801 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.094777 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12866e2-730f-4719-b736-a337032baa17-kube-api-access-h8jvs\" (UniqueName: \"kubernetes.io/projected/a12866e2-730f-4719-b736-a337032baa17-kube-api-access-h8jvs\") pod \"a12866e2-730f-4719-b736-a337032baa17\" (UID: \"a12866e2-730f-4719-b736-a337032baa17\") " Nov 03 07:36:09.094863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.094848 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-plugins-config-volume\") pod \"c96afb9c-3e74-479f-b1d2-f8dfce47b70e\" (UID: \"c96afb9c-3e74-479f-b1d2-f8dfce47b70e\") " Nov 03 07:36:09.095066 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.095050 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a12866e2-730f-4719-b736-a337032baa17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12866e2-730f-4719-b736-a337032baa17-plugins-config-volume\") pod \"a12866e2-730f-4719-b736-a337032baa17\" (UID: \"a12866e2-730f-4719-b736-a337032baa17\") " Nov 03 07:36:09.095416 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.095377 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-plugins-config-volume" pod "c96afb9c-3e74-479f-b1d2-f8dfce47b70e" (UID: "c96afb9c-3e74-479f-b1d2-f8dfce47b70e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:36:09.095416 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.095409 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12866e2-730f-4719-b736-a337032baa17-plugins-config-volume" pod "a12866e2-730f-4719-b736-a337032baa17" (UID: "a12866e2-730f-4719-b736-a337032baa17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:36:09.095643 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.095624 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12866e2-730f-4719-b736-a337032baa17-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:09.095704 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.095651 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:09.096916 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.096896 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-kube-api-access-v6hcb" pod "c96afb9c-3e74-479f-b1d2-f8dfce47b70e" (UID: "c96afb9c-3e74-479f-b1d2-f8dfce47b70e"). InnerVolumeSpecName "kube-api-access-v6hcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:36:09.096999 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.096950 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12866e2-730f-4719-b736-a337032baa17-kube-api-access-h8jvs" pod "a12866e2-730f-4719-b736-a337032baa17" (UID: "a12866e2-730f-4719-b736-a337032baa17"). InnerVolumeSpecName "kube-api-access-h8jvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:36:09.197342 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.197261 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6hcb\" (UniqueName: \"kubernetes.io/projected/c96afb9c-3e74-479f-b1d2-f8dfce47b70e-kube-api-access-v6hcb\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:09.197342 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.197291 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8jvs\" (UniqueName: \"kubernetes.io/projected/a12866e2-730f-4719-b736-a337032baa17-kube-api-access-h8jvs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:09.282470 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.282432 4409 generic.go:356] "Generic (PLEG): container finished" podID="c96afb9c-3e74-479f-b1d2-f8dfce47b70e" containerID="a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d" exitCode=0 Nov 03 07:36:09.282629 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.282497 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" Nov 03 07:36:09.282629 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.282519 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" event={"ID":"c96afb9c-3e74-479f-b1d2-f8dfce47b70e","Type":"ContainerDied","Data":"a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d"} Nov 03 07:36:09.282629 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.282614 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk" event={"ID":"c96afb9c-3e74-479f-b1d2-f8dfce47b70e","Type":"ContainerDied","Data":"d3972dcdaf26a6c3fbf7d7053cd5df5900795db356c39c436343460a390cf11c"} Nov 03 07:36:09.282876 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.282646 4409 scope.go:122] "RemoveContainer" containerID="a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d" Nov 03 07:36:09.284305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.284277 4409 generic.go:356] "Generic (PLEG): container finished" podID="a12866e2-730f-4719-b736-a337032baa17" containerID="163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461" exitCode=0 Nov 03 07:36:09.284643 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.284611 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" event={"ID":"a12866e2-730f-4719-b736-a337032baa17","Type":"ContainerDied","Data":"163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461"} Nov 03 07:36:09.284776 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.284656 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" event={"ID":"a12866e2-730f-4719-b736-a337032baa17","Type":"ContainerDied","Data":"d6937bcc832b17524db3d7cf1e4fc0ef6a4dcf72dd8b90fb6e1ec6cb5c67baae"} Nov 03 07:36:09.284857 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.284772 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8" Nov 03 07:36:09.295456 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.295435 4409 scope.go:122] "RemoveContainer" containerID="a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d" Nov 03 07:36:09.295798 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:36:09.295776 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d\": container with ID starting with a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d not found: ID does not exist" containerID="a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d" Nov 03 07:36:09.295874 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.295811 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d"} err="failed to get container status \"a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d\": rpc error: code = NotFound desc = could not find container \"a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d\": container with ID starting with a79e3d96f3a664ac3afe577256fff9f84f7c735110292d12809a9ad93f25ae5d not found: ID does not exist" Nov 03 07:36:09.295874 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.295829 4409 scope.go:122] "RemoveContainer" containerID="163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461" Nov 03 07:36:09.305121 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.305105 4409 scope.go:122] "RemoveContainer" containerID="163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461" Nov 03 07:36:09.305377 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:36:09.305360 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461\": container with ID starting with 163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461 not found: ID does not exist" containerID="163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461" Nov 03 07:36:09.305441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.305381 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461"} err="failed to get container status \"163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461\": rpc error: code = NotFound desc = could not find container \"163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461\": container with ID starting with 163f6e6fb95321c0995ad5bd6e536f30567e97ad600523bf5a3f72c43e223461 not found: ID does not exist" Nov 03 07:36:09.451405 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.451337 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk"] Nov 03 07:36:09.456786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.456761 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e6dc7-5d79cddb47-xb7xk"] Nov 03 07:36:09.572754 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.568967 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8"] Nov 03 07:36:09.572754 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:09.571341 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8e6dc7-74d68d8fcc-bvkp8"] Nov 03 07:36:10.315004 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.314973 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk"] Nov 03 07:36:10.315597 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315583 4409 cpu_manager.go:414] "containerMap: removing container" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" Nov 03 07:36:10.315597 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315598 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" Nov 03 07:36:10.315673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315611 4409 cpu_manager.go:414] "containerMap: removing container" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerName="istio-proxy" Nov 03 07:36:10.315673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315617 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerName="istio-proxy" Nov 03 07:36:10.315673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315629 4409 cpu_manager.go:414] "containerMap: removing container" podUID="1d20a97a-9ef7-4e1a-b97e-5d420f609b59" containerName="payload-pre-processing" Nov 03 07:36:10.315673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315634 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d20a97a-9ef7-4e1a-b97e-5d420f609b59" containerName="payload-pre-processing" Nov 03 07:36:10.315673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315642 4409 cpu_manager.go:414] "containerMap: removing container" podUID="a12866e2-730f-4719-b736-a337032baa17" containerName="payload-processing" Nov 03 07:36:10.315673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315648 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a12866e2-730f-4719-b736-a337032baa17" containerName="payload-processing" Nov 03 07:36:10.315673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315657 4409 cpu_manager.go:414] "containerMap: removing container" podUID="c96afb9c-3e74-479f-b1d2-f8dfce47b70e" containerName="payload-pre-processing" Nov 03 07:36:10.315673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315661 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c96afb9c-3e74-479f-b1d2-f8dfce47b70e" containerName="payload-pre-processing" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315680 4409 cpu_manager.go:414] "containerMap: removing container" podUID="0d7f5b7f-0ac7-40d4-be81-cd09336f25b8" containerName="maas-api" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315684 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7f5b7f-0ac7-40d4-be81-cd09336f25b8" containerName="maas-api" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315698 4409 cpu_manager.go:414] "containerMap: removing container" podUID="01edfa30-1ac5-411a-a084-933f1a3578dd" containerName="maas-api" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315702 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01edfa30-1ac5-411a-a084-933f1a3578dd" containerName="maas-api" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315731 4409 cpu_manager.go:414] "containerMap: removing container" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315736 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315746 4409 cpu_manager.go:414] "containerMap: removing container" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerName="istio-proxy" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315750 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerName="istio-proxy" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315843 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="c96afb9c-3e74-479f-b1d2-f8dfce47b70e" containerName="payload-pre-processing" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315855 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7f5b7f-0ac7-40d4-be81-cd09336f25b8" containerName="maas-api" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315864 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="01edfa30-1ac5-411a-a084-933f1a3578dd" containerName="maas-api" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315869 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c96119-0bed-477e-9992-eb73a9761fca" containerName="istio-proxy" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315875 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315880 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerName="istio-proxy" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315886 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315892 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d20a97a-9ef7-4e1a-b97e-5d420f609b59" containerName="payload-pre-processing" Nov 03 07:36:10.315949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.315897 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="a12866e2-730f-4719-b736-a337032baa17" containerName="payload-processing" Nov 03 07:36:10.321576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.321557 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:10.378569 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.378545 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk"] Nov 03 07:36:10.413534 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.413507 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-maas-api-tls\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:10.413681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.413550 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxn5z\" (UniqueName: \"kubernetes.io/projected/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-kube-api-access-pxn5z\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:10.413681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.413583 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:10.413803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.413693 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-metrics-certs\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:10.423590 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.423564 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0e8bf4-serving-cert\"" Nov 03 07:36:10.514629 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.514598 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-maas-api-tls\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:10.514803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.514643 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxn5z\" (UniqueName: \"kubernetes.io/projected/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-kube-api-access-pxn5z\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:10.514803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.514669 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:10.514803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.514697 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-metrics-certs\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:10.514803 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:36:10.514771 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-serving-cert: secret "maas-api-e2e-inf-a-0e8bf4-serving-cert" not found Nov 03 07:36:10.514977 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:36:10.514846 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-maas-api-tls podName:24e61dd8-0436-41bd-b80e-bdfd1f70b16b nodeName:}" failed. No retries permitted until 2026-11-03 07:36:11.014823289 +0000 UTC m=+2226.207365031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-maas-api-tls") pod "maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" (UID: "24e61dd8-0436-41bd-b80e-bdfd1f70b16b") : secret "maas-api-e2e-inf-a-0e8bf4-serving-cert" not found Nov 03 07:36:10.515369 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.515341 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:10.517236 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.517217 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-metrics-certs\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:10.578894 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.578838 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxn5z\" (UniqueName: \"kubernetes.io/projected/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-kube-api-access-pxn5z\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:10.914156 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.914083 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a12866e2-730f-4719-b736-a337032baa17" path="/var/lib/kubelet/pods/a12866e2-730f-4719-b736-a337032baa17/volumes" Nov 03 07:36:10.914492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:10.914479 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c96afb9c-3e74-479f-b1d2-f8dfce47b70e" path="/var/lib/kubelet/pods/c96afb9c-3e74-479f-b1d2-f8dfce47b70e/volumes" Nov 03 07:36:11.032077 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:11.032048 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-maas-api-tls\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:11.034674 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:11.034649 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-maas-api-tls\") pod \"maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:11.241453 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:11.241381 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:11.628836 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:11.628812 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk"] Nov 03 07:36:11.630087 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:36:11.630061 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e61dd8_0436_41bd_b80e_bdfd1f70b16b.slice/crio-437095b04ab566c7ccdfc5c7f388b9ce85c444e058620829a05c72f636957502 WatchSource:0}: Error finding container 437095b04ab566c7ccdfc5c7f388b9ce85c444e058620829a05c72f636957502: Status 404 returned error can't find the container with id 437095b04ab566c7ccdfc5c7f388b9ce85c444e058620829a05c72f636957502 Nov 03 07:36:12.311280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:12.311239 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" event={"ID":"24e61dd8-0436-41bd-b80e-bdfd1f70b16b","Type":"ContainerStarted","Data":"437095b04ab566c7ccdfc5c7f388b9ce85c444e058620829a05c72f636957502"} Nov 03 07:36:13.320031 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:13.319994 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" event={"ID":"24e61dd8-0436-41bd-b80e-bdfd1f70b16b","Type":"ContainerStarted","Data":"e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25"} Nov 03 07:36:13.320502 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:13.320112 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:13.442022 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:13.441963 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" podStartSLOduration=2.5032657609999998 podStartE2EDuration="3.441945201s" totalImagesPullingTime="938.67944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:36:13.44081623 +0000 UTC m=+2228.633358053" watchObservedRunningTime="2026-11-03 07:36:13.441945201 +0000 UTC m=+2228.634486953" Nov 03 07:36:19.333875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:19.333842 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:36:30.658363 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.658331 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6"] Nov 03 07:36:30.659176 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.659153 4409 cpu_manager.go:414] "containerMap: removing container" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" Nov 03 07:36:30.659176 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.659176 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc8eec3-9498-4244-afe5-456b9989e5f9" containerName="istio-proxy" Nov 03 07:36:30.659332 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.659297 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="7449c7e0-cb8b-47d0-9cd3-7bcd789b5a91" containerName="istio-proxy" Nov 03 07:36:30.662985 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.662962 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:30.717499 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.717471 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-295fba7c-serving-cert\"" Nov 03 07:36:30.769346 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.769312 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6"] Nov 03 07:36:30.845444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.845415 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-maas-api-tls\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:30.845599 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.845562 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9q4\" (UniqueName: \"kubernetes.io/projected/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-kube-api-access-nf9q4\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:30.845643 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.845635 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-metrics-certs\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:30.845723 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.845692 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-openshift-service-ca\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:30.947869 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.947793 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-maas-api-tls\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:30.948028 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.947932 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9q4\" (UniqueName: \"kubernetes.io/projected/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-kube-api-access-nf9q4\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:30.948098 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.948020 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-metrics-certs\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:30.948098 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.948081 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-openshift-service-ca\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:30.948746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.948726 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-openshift-service-ca\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:30.950379 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.950348 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-maas-api-tls\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:30.950775 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:30.950749 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-metrics-certs\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:31.010911 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:31.010884 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9q4\" (UniqueName: \"kubernetes.io/projected/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-kube-api-access-nf9q4\") pod \"maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:31.276539 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:31.276452 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:31.660127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:31.660097 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6"] Nov 03 07:36:31.661215 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:36:31.661185 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d55a46_9c7a_4cc7_989a_a27c45ee3e91.slice/crio-46066e20ce4126bcba79a430b2e9aedac5d4525890c91d98dbc1a529bac119de WatchSource:0}: Error finding container 46066e20ce4126bcba79a430b2e9aedac5d4525890c91d98dbc1a529bac119de: Status 404 returned error can't find the container with id 46066e20ce4126bcba79a430b2e9aedac5d4525890c91d98dbc1a529bac119de Nov 03 07:36:32.448458 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:32.448424 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" event={"ID":"17d55a46-9c7a-4cc7-989a-a27c45ee3e91","Type":"ContainerStarted","Data":"46066e20ce4126bcba79a430b2e9aedac5d4525890c91d98dbc1a529bac119de"} Nov 03 07:36:33.457579 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:33.457544 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" event={"ID":"17d55a46-9c7a-4cc7-989a-a27c45ee3e91","Type":"ContainerStarted","Data":"2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000"} Nov 03 07:36:33.458050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:33.457742 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:33.579377 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:33.579319 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" podStartSLOduration=2.779232977 podStartE2EDuration="3.579303904s" totalImagesPullingTime="800.070927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:36:33.578097839 +0000 UTC m=+2248.770639590" watchObservedRunningTime="2026-11-03 07:36:33.579303904 +0000 UTC m=+2248.771845654" Nov 03 07:36:38.776178 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:36:38.776141 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:36:39.469582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:39.469551 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:36:45.273262 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.273220 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4"] Nov 03 07:36:45.273680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.273505 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" podUID="bac1b906-cbe4-4b96-9b01-dd71f6c0dacc" containerName="maas-api" containerID="cri-o://4da62902c89e1e1fabea1f7cc917133f44f986dfeae65597e68c1a17f333060b" gracePeriod=30 Nov 03 07:36:45.556060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.555856 4409 generic.go:356] "Generic (PLEG): container finished" podID="bac1b906-cbe4-4b96-9b01-dd71f6c0dacc" containerID="4da62902c89e1e1fabea1f7cc917133f44f986dfeae65597e68c1a17f333060b" exitCode=0 Nov 03 07:36:45.556060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.555984 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" event={"ID":"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc","Type":"ContainerDied","Data":"4da62902c89e1e1fabea1f7cc917133f44f986dfeae65597e68c1a17f333060b"} Nov 03 07:36:45.604618 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.604590 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:36:45.770657 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.770623 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-metrics-certs\") pod \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " Nov 03 07:36:45.770878 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.770701 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-maas-api-tls\") pod \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " Nov 03 07:36:45.770878 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.770870 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-kube-api-access-jlvqw\" (UniqueName: \"kubernetes.io/projected/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-kube-api-access-jlvqw\") pod \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " Nov 03 07:36:45.771001 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.770930 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-openshift-service-ca\") pod \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\" (UID: \"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc\") " Nov 03 07:36:45.771310 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.771284 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-openshift-service-ca" pod "bac1b906-cbe4-4b96-9b01-dd71f6c0dacc" (UID: "bac1b906-cbe4-4b96-9b01-dd71f6c0dacc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:36:45.771954 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.771932 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:45.773132 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.773099 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-metrics-certs" pod "bac1b906-cbe4-4b96-9b01-dd71f6c0dacc" (UID: "bac1b906-cbe4-4b96-9b01-dd71f6c0dacc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:36:45.773223 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.773179 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-maas-api-tls" pod "bac1b906-cbe4-4b96-9b01-dd71f6c0dacc" (UID: "bac1b906-cbe4-4b96-9b01-dd71f6c0dacc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:36:45.773223 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.773182 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-kube-api-access-jlvqw" pod "bac1b906-cbe4-4b96-9b01-dd71f6c0dacc" (UID: "bac1b906-cbe4-4b96-9b01-dd71f6c0dacc"). InnerVolumeSpecName "kube-api-access-jlvqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:36:45.885786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.885298 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:45.885786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.885342 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:45.885786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:45.885358 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlvqw\" (UniqueName: \"kubernetes.io/projected/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc-kube-api-access-jlvqw\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:46.563852 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:46.563826 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" Nov 03 07:36:46.563852 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:46.563835 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4" event={"ID":"bac1b906-cbe4-4b96-9b01-dd71f6c0dacc","Type":"ContainerDied","Data":"7dbfe5ac9a640b1c236574c39114edede68cc5165b21446fc10996645a540405"} Nov 03 07:36:46.564345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:46.563899 4409 scope.go:122] "RemoveContainer" containerID="4da62902c89e1e1fabea1f7cc917133f44f986dfeae65597e68c1a17f333060b" Nov 03 07:36:46.736906 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:46.736870 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4"] Nov 03 07:36:46.785288 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:46.785257 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-893d83-58dc57fd88-25tr4"] Nov 03 07:36:46.914010 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:46.913930 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac1b906-cbe4-4b96-9b01-dd71f6c0dacc" path="/var/lib/kubelet/pods/bac1b906-cbe4-4b96-9b01-dd71f6c0dacc/volumes" Nov 03 07:36:56.367276 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:56.367236 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf"] Nov 03 07:36:56.369676 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:56.367477 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" containerID="cri-o://6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c" gracePeriod=30 Nov 03 07:36:58.670679 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:58.670644 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr"] Nov 03 07:36:58.671099 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:58.670887 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" podUID="fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7" containerName="maas-api" containerID="cri-o://56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee" gracePeriod=30 Nov 03 07:36:58.960792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:58.954074 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:36:59.083677 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.083637 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-kube-api-access-xc8vb\" (UniqueName: \"kubernetes.io/projected/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-kube-api-access-xc8vb\") pod \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " Nov 03 07:36:59.083919 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.083763 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-maas-api-tls\") pod \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " Nov 03 07:36:59.084545 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.084501 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-metrics-certs\") pod \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " Nov 03 07:36:59.084866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.084844 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-openshift-service-ca\") pod \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\" (UID: \"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7\") " Nov 03 07:36:59.086006 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.085981 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-openshift-service-ca" pod "fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7" (UID: "fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:36:59.089784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.089704 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-metrics-certs" pod "fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7" (UID: "fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:36:59.090374 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.090336 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-maas-api-tls" pod "fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7" (UID: "fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:36:59.090911 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.090889 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-kube-api-access-xc8vb" pod "fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7" (UID: "fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7"). InnerVolumeSpecName "kube-api-access-xc8vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:36:59.186638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.186612 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc8vb\" (UniqueName: \"kubernetes.io/projected/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-kube-api-access-xc8vb\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:59.186796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.186641 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:59.186796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.186655 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:59.186796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.186668 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:36:59.654748 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.654697 4409 generic.go:356] "Generic (PLEG): container finished" podID="fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7" containerID="56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee" exitCode=0 Nov 03 07:36:59.654948 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.654765 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" Nov 03 07:36:59.654948 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.654784 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" event={"ID":"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7","Type":"ContainerDied","Data":"56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee"} Nov 03 07:36:59.654948 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.654846 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr" event={"ID":"fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7","Type":"ContainerDied","Data":"7b6f3580fdcba37093bf678947d78078d522260afe91004cb6a654afab11e654"} Nov 03 07:36:59.654948 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.654870 4409 scope.go:122] "RemoveContainer" containerID="56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee" Nov 03 07:36:59.665499 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.665477 4409 scope.go:122] "RemoveContainer" containerID="56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee" Nov 03 07:36:59.665786 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:36:59.665763 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee\": container with ID starting with 56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee not found: ID does not exist" containerID="56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee" Nov 03 07:36:59.665865 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.665795 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee"} err="failed to get container status \"56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee\": rpc error: code = NotFound desc = could not find container \"56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee\": container with ID starting with 56edffeb9ca4c154fcf5335d729610fe77c435a8f9244665b2181430d20830ee not found: ID does not exist" Nov 03 07:36:59.831929 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.831899 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr"] Nov 03 07:36:59.879352 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:36:59.879322 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1fa30f-59685857f5-88rxr"] Nov 03 07:37:00.913664 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:00.913627 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7" path="/var/lib/kubelet/pods/fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7/volumes" Nov 03 07:37:01.655462 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.655436 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:37:01.670974 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.670944 4409 generic.go:356] "Generic (PLEG): container finished" podID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerID="6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c" exitCode=0 Nov 03 07:37:01.671092 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.670991 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" Nov 03 07:37:01.671092 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.671026 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" event={"ID":"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6","Type":"ContainerDied","Data":"6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c"} Nov 03 07:37:01.671166 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.671098 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf" event={"ID":"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6","Type":"ContainerDied","Data":"24532d676d93904f188ce3eb1443b284a030b8f6e729081963a84eb2ff2a8b17"} Nov 03 07:37:01.671166 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.671120 4409 scope.go:122] "RemoveContainer" containerID="6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c" Nov 03 07:37:01.681056 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.681038 4409 scope.go:122] "RemoveContainer" containerID="b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd" Nov 03 07:37:01.690136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.690122 4409 scope.go:122] "RemoveContainer" containerID="6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c" Nov 03 07:37:01.690390 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:37:01.690373 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c\": container with ID starting with 6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c not found: ID does not exist" containerID="6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c" Nov 03 07:37:01.690447 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.690398 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c"} err="failed to get container status \"6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c\": rpc error: code = NotFound desc = could not find container \"6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c\": container with ID starting with 6b9690122ce60fdd99fc97f495ceb7c5f5bab970b4cd4cba282f642e9418809c not found: ID does not exist" Nov 03 07:37:01.690447 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.690413 4409 scope.go:122] "RemoveContainer" containerID="b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd" Nov 03 07:37:01.690628 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:37:01.690612 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd\": container with ID starting with b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd not found: ID does not exist" containerID="b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd" Nov 03 07:37:01.690669 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.690632 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd"} err="failed to get container status \"b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd\": rpc error: code = NotFound desc = could not find container \"b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd\": container with ID starting with b84d7cb54fe70b301b9dffd89459247b63ae0a9ac55cb83dffee7d04ab1c4ccd not found: ID does not exist" Nov 03 07:37:01.717240 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.717215 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istiod-ca-cert\") pod \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " Nov 03 07:37:01.717415 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.717320 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-token\" (UniqueName: \"kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-token\") pod \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " Nov 03 07:37:01.717415 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.717357 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-envoy\") pod \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " Nov 03 07:37:01.717509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.717492 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-kube-api-access-tndrm\" (UniqueName: \"kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-kube-api-access-tndrm\") pod \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " Nov 03 07:37:01.717546 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.717526 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-podinfo\") pod \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " Nov 03 07:37:01.717595 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.717577 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istiod-ca-cert" pod "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" (UID: "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:01.717595 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.717588 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-data\") pod \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " Nov 03 07:37:01.717734 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.717621 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-certs\") pod \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " Nov 03 07:37:01.717734 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.717675 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-credential-socket\") pod \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " Nov 03 07:37:01.717861 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.717839 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-socket\") pod \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\" (UID: \"dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6\") " Nov 03 07:37:01.717951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.717917 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-data" pod "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" (UID: "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:01.718011 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.717997 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-certs" pod "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" (UID: "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:01.718158 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.718134 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-credential-socket" pod "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" (UID: "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:01.718211 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.718160 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-socket" pod "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" (UID: "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:01.718665 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.718645 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:01.718763 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.718675 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:01.718763 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.718692 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:01.718763 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.718707 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:01.718763 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.718740 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:01.719852 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.719824 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-kube-api-access-tndrm" pod "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" (UID: "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6"). InnerVolumeSpecName "kube-api-access-tndrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:01.720005 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.719986 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-envoy" pod "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" (UID: "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:37:01.720150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.720132 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-token" pod "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" (UID: "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:01.720221 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.720202 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-podinfo" pod "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" (UID: "dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:37:01.820136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.820113 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:01.820136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.820136 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:01.820289 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.820147 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tndrm\" (UniqueName: \"kubernetes.io/projected/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-kube-api-access-tndrm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:01.820289 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:01.820157 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:02.096609 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:02.096579 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf"] Nov 03 07:37:02.102798 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:02.102767 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-893d83-openshift-default-6b47fc65dc-nd6pf"] Nov 03 07:37:02.914136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:02.914105 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" path="/var/lib/kubelet/pods/dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6/volumes" Nov 03 07:37:05.537743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:05.535328 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn"] Nov 03 07:37:05.537743 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:05.535565 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" podUID="261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46" containerName="maas-api" containerID="cri-o://0e8a7987fed62ed82d2381c31cc63ebb126e6edf8736a5e566c999828ea219eb" gracePeriod=30 Nov 03 07:37:05.738770 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:05.727029 4409 generic.go:356] "Generic (PLEG): container finished" podID="261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46" containerID="0e8a7987fed62ed82d2381c31cc63ebb126e6edf8736a5e566c999828ea219eb" exitCode=0 Nov 03 07:37:05.738770 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:05.727229 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" event={"ID":"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46","Type":"ContainerDied","Data":"0e8a7987fed62ed82d2381c31cc63ebb126e6edf8736a5e566c999828ea219eb"} Nov 03 07:37:06.162811 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.162785 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:37:06.188738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.188254 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-metrics-certs\") pod \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " Nov 03 07:37:06.188738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.188346 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-kube-api-access-296gr\" (UniqueName: \"kubernetes.io/projected/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-kube-api-access-296gr\") pod \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " Nov 03 07:37:06.188738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.188474 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-maas-api-tls\") pod \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " Nov 03 07:37:06.188738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.188516 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-openshift-service-ca\") pod \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\" (UID: \"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46\") " Nov 03 07:37:06.196545 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.196516 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-openshift-service-ca" pod "261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46" (UID: "261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:06.198820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.198784 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-kube-api-access-296gr" pod "261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46" (UID: "261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46"). InnerVolumeSpecName "kube-api-access-296gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:06.199234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.199198 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-maas-api-tls" pod "261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46" (UID: "261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:37:06.199629 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.199602 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-metrics-certs" pod "261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46" (UID: "261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:37:06.291521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.290699 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:06.291521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.290752 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:06.291521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.290768 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:06.291521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.290783 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-296gr\" (UniqueName: \"kubernetes.io/projected/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46-kube-api-access-296gr\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:06.735888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.735862 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" Nov 03 07:37:06.735888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.735872 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn" event={"ID":"261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46","Type":"ContainerDied","Data":"5feb3a5a4c884cbbe5ce450604e60ec99f899d818b5b8ddba36752bca51b3273"} Nov 03 07:37:06.736415 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.735936 4409 scope.go:122] "RemoveContainer" containerID="0e8a7987fed62ed82d2381c31cc63ebb126e6edf8736a5e566c999828ea219eb" Nov 03 07:37:06.790651 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.790615 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g"] Nov 03 07:37:06.792982 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.792934 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" podUID="7e84d830-9ae7-41f5-a105-068e1d316171" containerName="maas-api" containerID="cri-o://d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e" gracePeriod=30 Nov 03 07:37:06.921116 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.918867 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn"] Nov 03 07:37:06.921116 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:06.920362 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e6dc7-6fcdd44bcb-wrkmn"] Nov 03 07:37:07.280791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.265941 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:37:07.354106 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.354068 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-maas-api-tls\") pod \"7e84d830-9ae7-41f5-a105-068e1d316171\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " Nov 03 07:37:07.354281 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.354161 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e84d830-9ae7-41f5-a105-068e1d316171-kube-api-access-msvlc\" (UniqueName: \"kubernetes.io/projected/7e84d830-9ae7-41f5-a105-068e1d316171-kube-api-access-msvlc\") pod \"7e84d830-9ae7-41f5-a105-068e1d316171\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " Nov 03 07:37:07.354337 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.354314 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e84d830-9ae7-41f5-a105-068e1d316171-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e84d830-9ae7-41f5-a105-068e1d316171-openshift-service-ca\") pod \"7e84d830-9ae7-41f5-a105-068e1d316171\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " Nov 03 07:37:07.354392 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.354377 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-metrics-certs\") pod \"7e84d830-9ae7-41f5-a105-068e1d316171\" (UID: \"7e84d830-9ae7-41f5-a105-068e1d316171\") " Nov 03 07:37:07.354779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.354750 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e84d830-9ae7-41f5-a105-068e1d316171-openshift-service-ca" pod "7e84d830-9ae7-41f5-a105-068e1d316171" (UID: "7e84d830-9ae7-41f5-a105-068e1d316171"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:07.355206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.355182 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e84d830-9ae7-41f5-a105-068e1d316171-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.356396 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.356369 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-maas-api-tls" pod "7e84d830-9ae7-41f5-a105-068e1d316171" (UID: "7e84d830-9ae7-41f5-a105-068e1d316171"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:37:07.356494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.356451 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-metrics-certs" pod "7e84d830-9ae7-41f5-a105-068e1d316171" (UID: "7e84d830-9ae7-41f5-a105-068e1d316171"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:37:07.356611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.356597 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e84d830-9ae7-41f5-a105-068e1d316171-kube-api-access-msvlc" pod "7e84d830-9ae7-41f5-a105-068e1d316171" (UID: "7e84d830-9ae7-41f5-a105-068e1d316171"). InnerVolumeSpecName "kube-api-access-msvlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:07.456886 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.456853 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.456886 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.456880 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e84d830-9ae7-41f5-a105-068e1d316171-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.456886 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.456891 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msvlc\" (UniqueName: \"kubernetes.io/projected/7e84d830-9ae7-41f5-a105-068e1d316171-kube-api-access-msvlc\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:07.743866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.743834 4409 generic.go:356] "Generic (PLEG): container finished" podID="7e84d830-9ae7-41f5-a105-068e1d316171" containerID="d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e" exitCode=0 Nov 03 07:37:07.744245 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.743877 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" Nov 03 07:37:07.744245 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.743876 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" event={"ID":"7e84d830-9ae7-41f5-a105-068e1d316171","Type":"ContainerDied","Data":"d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e"} Nov 03 07:37:07.744245 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.743927 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g" event={"ID":"7e84d830-9ae7-41f5-a105-068e1d316171","Type":"ContainerDied","Data":"f51485b4222291919e4be6a14e68f334f051b131221274da2cbbe18b5f3713b9"} Nov 03 07:37:07.744245 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.743949 4409 scope.go:122] "RemoveContainer" containerID="d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e" Nov 03 07:37:07.756124 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.756107 4409 scope.go:122] "RemoveContainer" containerID="d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e" Nov 03 07:37:07.756403 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:37:07.756387 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e\": container with ID starting with d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e not found: ID does not exist" containerID="d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e" Nov 03 07:37:07.756477 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.756415 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e"} err="failed to get container status \"d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e\": rpc error: code = NotFound desc = could not find container \"d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e\": container with ID starting with d4db680673460055632929b7a9f9308327cdb9b7419b68d34b695af63848d51e not found: ID does not exist" Nov 03 07:37:07.938640 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.938600 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g"] Nov 03 07:37:07.942577 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:07.942549 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6d9a2-5867d45577-wr22g"] Nov 03 07:37:08.801329 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:37:08.801287 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:37:08.914434 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:08.914393 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46" path="/var/lib/kubelet/pods/261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46/volumes" Nov 03 07:37:08.914961 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:08.914940 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e84d830-9ae7-41f5-a105-068e1d316171" path="/var/lib/kubelet/pods/7e84d830-9ae7-41f5-a105-068e1d316171/volumes" Nov 03 07:37:09.625169 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:09.625133 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2"] Nov 03 07:37:09.741592 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:09.741558 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf"] Nov 03 07:37:11.284925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.284891 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc"] Nov 03 07:37:11.286343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286308 4409 cpu_manager.go:414] "containerMap: removing container" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" Nov 03 07:37:11.286343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286337 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" Nov 03 07:37:11.286507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286366 4409 cpu_manager.go:414] "containerMap: removing container" podUID="bac1b906-cbe4-4b96-9b01-dd71f6c0dacc" containerName="maas-api" Nov 03 07:37:11.286507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286376 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac1b906-cbe4-4b96-9b01-dd71f6c0dacc" containerName="maas-api" Nov 03 07:37:11.286507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286391 4409 cpu_manager.go:414] "containerMap: removing container" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" Nov 03 07:37:11.286507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286398 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" Nov 03 07:37:11.286507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286428 4409 cpu_manager.go:414] "containerMap: removing container" podUID="fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7" containerName="maas-api" Nov 03 07:37:11.286507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286436 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7" containerName="maas-api" Nov 03 07:37:11.286507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286450 4409 cpu_manager.go:414] "containerMap: removing container" podUID="7e84d830-9ae7-41f5-a105-068e1d316171" containerName="maas-api" Nov 03 07:37:11.286507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286457 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e84d830-9ae7-41f5-a105-068e1d316171" containerName="maas-api" Nov 03 07:37:11.286507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286467 4409 cpu_manager.go:414] "containerMap: removing container" podUID="261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46" containerName="maas-api" Nov 03 07:37:11.286507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286475 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46" containerName="maas-api" Nov 03 07:37:11.286905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286638 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" Nov 03 07:37:11.286905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286655 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac1b906-cbe4-4b96-9b01-dd71f6c0dacc" containerName="maas-api" Nov 03 07:37:11.286905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286666 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc24a90a-dbe9-45f9-bdfc-e09fbfd5b9d6" containerName="istio-proxy" Nov 03 07:37:11.286905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286679 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb61af4a-ab5e-4ad1-85bc-b50a300bd9a7" containerName="maas-api" Nov 03 07:37:11.286905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286689 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e84d830-9ae7-41f5-a105-068e1d316171" containerName="maas-api" Nov 03 07:37:11.286905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.286699 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="261ffc0f-a6a3-4504-9cf4-ed54bb1e5e46" containerName="maas-api" Nov 03 07:37:11.292142 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.292119 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.350162 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.350128 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc"] Nov 03 07:37:11.436343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.436314 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a8983d-openshift-default-dockercfg-j8zgh\"" Nov 03 07:37:11.479747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.479038 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-token\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.479747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.479121 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g667\" (UniqueName: \"kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-kube-api-access-8g667\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.479747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.479151 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-envoy\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.479747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.479337 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-socket\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.479747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.479388 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-data\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.479747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.479423 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-credential-socket\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.479747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.479448 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-podinfo\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.479747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.479499 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istiod-ca-cert\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.479747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.479531 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-certs\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.580629 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.580595 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-envoy\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.580868 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.580759 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-socket\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.580868 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.580799 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-data\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.580868 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.580824 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-credential-socket\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.580868 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.580841 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-podinfo\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.581109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.580885 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istiod-ca-cert\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.581109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.580919 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-certs\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.581109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.580975 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-token\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.581109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.581038 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g667\" (UniqueName: \"kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-kube-api-access-8g667\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.581281 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.581159 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-socket\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.581343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.581327 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-credential-socket\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.581401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.581355 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-certs\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.581459 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.581401 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-data\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.581666 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.581643 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istiod-ca-cert\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.583401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.583382 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-envoy\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.583630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.583609 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-podinfo\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.643925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.643891 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-token\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.644098 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.644078 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g667\" (UniqueName: \"kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-kube-api-access-8g667\") pod \"e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:11.907304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:11.907221 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:12.345257 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:12.345232 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc"] Nov 03 07:37:12.345968 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:37:12.345945 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86da6dc_a180_4dd3_86e0_b34c28b47c37.slice/crio-5b1f2a733efe3c53de8a06a6dae065b4ccf96a82302ab37b2701b507c06c037d WatchSource:0}: Error finding container 5b1f2a733efe3c53de8a06a6dae065b4ccf96a82302ab37b2701b507c06c037d: Status 404 returned error can't find the container with id 5b1f2a733efe3c53de8a06a6dae065b4ccf96a82302ab37b2701b507c06c037d Nov 03 07:37:12.348384 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:12.348352 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:12.348463 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:12.348415 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:12.348463 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:12.348444 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:12.781724 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:12.781615 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" event={"ID":"d86da6dc-a180-4dd3-86e0-b34c28b47c37","Type":"ContainerStarted","Data":"9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e"} Nov 03 07:37:12.781724 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:12.781675 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" event={"ID":"d86da6dc-a180-4dd3-86e0-b34c28b47c37","Type":"ContainerStarted","Data":"5b1f2a733efe3c53de8a06a6dae065b4ccf96a82302ab37b2701b507c06c037d"} Nov 03 07:37:12.904777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:12.904729 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" podStartSLOduration=1.904696678 podStartE2EDuration="1.904696678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:12.903407735 +0000 UTC m=+2288.095949495" watchObservedRunningTime="2026-11-03 07:37:12.904696678 +0000 UTC m=+2288.097238427" Nov 03 07:37:12.913996 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:12.913973 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:13.914118 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:13.914086 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:14.626155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:14.626120 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" podUID="5ad61f40-0c74-4ea0-9dbe-1361ea562cba" containerName="payload-processing" containerID="cri-o://a5975f9cfe011d0c6dfb5a4b72393dc2f062a4919fcd9d55342fb0e357cccb89" gracePeriod=25 Nov 03 07:37:14.732883 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:37:14.732843 4409 logging.go:55] [core] [Channel #4428 SubChannel #4430]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.114:50052", ServerName: "10.133.0.114:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.114:50052: connect: connection refused" Nov 03 07:37:14.744107 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:14.744072 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" podUID="090c672f-286d-4291-a953-3eddfdadbb15" containerName="payload-pre-processing" containerID="cri-o://81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a" gracePeriod=25 Nov 03 07:37:14.797595 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:14.797563 4409 generic.go:356] "Generic (PLEG): container finished" podID="5ad61f40-0c74-4ea0-9dbe-1361ea562cba" containerID="a5975f9cfe011d0c6dfb5a4b72393dc2f062a4919fcd9d55342fb0e357cccb89" exitCode=0 Nov 03 07:37:14.797746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:14.797637 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" event={"ID":"5ad61f40-0c74-4ea0-9dbe-1361ea562cba","Type":"ContainerDied","Data":"a5975f9cfe011d0c6dfb5a4b72393dc2f062a4919fcd9d55342fb0e357cccb89"} Nov 03 07:37:14.798049 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:14.798028 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:14.799199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:14.799180 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:37:14.907367 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:14.907341 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" Nov 03 07:37:14.925497 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:14.925380 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-kube-api-access-vjv47\" (UniqueName: \"kubernetes.io/projected/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-kube-api-access-vjv47\") pod \"5ad61f40-0c74-4ea0-9dbe-1361ea562cba\" (UID: \"5ad61f40-0c74-4ea0-9dbe-1361ea562cba\") " Nov 03 07:37:14.929056 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:14.928144 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-kube-api-access-vjv47" pod "5ad61f40-0c74-4ea0-9dbe-1361ea562cba" (UID: "5ad61f40-0c74-4ea0-9dbe-1361ea562cba"). InnerVolumeSpecName "kube-api-access-vjv47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:14.929056 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:14.928217 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-plugins-config-volume\") pod \"5ad61f40-0c74-4ea0-9dbe-1361ea562cba\" (UID: \"5ad61f40-0c74-4ea0-9dbe-1361ea562cba\") " Nov 03 07:37:14.929056 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:14.928844 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjv47\" (UniqueName: \"kubernetes.io/projected/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-kube-api-access-vjv47\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:14.929480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:14.929444 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-plugins-config-volume" pod "5ad61f40-0c74-4ea0-9dbe-1361ea562cba" (UID: "5ad61f40-0c74-4ea0-9dbe-1361ea562cba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:15.030177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.030145 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad61f40-0c74-4ea0-9dbe-1361ea562cba-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:15.036492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.036468 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" Nov 03 07:37:15.130686 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.130605 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090c672f-286d-4291-a953-3eddfdadbb15-kube-api-access-ltg9g\" (UniqueName: \"kubernetes.io/projected/090c672f-286d-4291-a953-3eddfdadbb15-kube-api-access-ltg9g\") pod \"090c672f-286d-4291-a953-3eddfdadbb15\" (UID: \"090c672f-286d-4291-a953-3eddfdadbb15\") " Nov 03 07:37:15.130953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.130930 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/090c672f-286d-4291-a953-3eddfdadbb15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090c672f-286d-4291-a953-3eddfdadbb15-plugins-config-volume\") pod \"090c672f-286d-4291-a953-3eddfdadbb15\" (UID: \"090c672f-286d-4291-a953-3eddfdadbb15\") " Nov 03 07:37:15.131360 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.131317 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090c672f-286d-4291-a953-3eddfdadbb15-plugins-config-volume" pod "090c672f-286d-4291-a953-3eddfdadbb15" (UID: "090c672f-286d-4291-a953-3eddfdadbb15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:15.131561 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.131538 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090c672f-286d-4291-a953-3eddfdadbb15-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:15.133128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.133104 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090c672f-286d-4291-a953-3eddfdadbb15-kube-api-access-ltg9g" pod "090c672f-286d-4291-a953-3eddfdadbb15" (UID: "090c672f-286d-4291-a953-3eddfdadbb15"). InnerVolumeSpecName "kube-api-access-ltg9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:15.232646 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.232613 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltg9g\" (UniqueName: \"kubernetes.io/projected/090c672f-286d-4291-a953-3eddfdadbb15-kube-api-access-ltg9g\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:15.732932 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.732884 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" podUID="5ad61f40-0c74-4ea0-9dbe-1361ea562cba" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.114:50052\" within 1s: context deadline exceeded" Nov 03 07:37:15.804833 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.804802 4409 generic.go:356] "Generic (PLEG): container finished" podID="090c672f-286d-4291-a953-3eddfdadbb15" containerID="81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a" exitCode=0 Nov 03 07:37:15.805024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.804863 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" Nov 03 07:37:15.805024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.804874 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" event={"ID":"090c672f-286d-4291-a953-3eddfdadbb15","Type":"ContainerDied","Data":"81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a"} Nov 03 07:37:15.805024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.804924 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf" event={"ID":"090c672f-286d-4291-a953-3eddfdadbb15","Type":"ContainerDied","Data":"62cc4b284abf9c478bdad8d32f6f73385ade90e5049c9ae7957e929351c0c23e"} Nov 03 07:37:15.805024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.804946 4409 scope.go:122] "RemoveContainer" containerID="81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a" Nov 03 07:37:15.806496 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.806479 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" Nov 03 07:37:15.806609 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.806521 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2" event={"ID":"5ad61f40-0c74-4ea0-9dbe-1361ea562cba","Type":"ContainerDied","Data":"e89f41cf800d2c25a252a4a77d67e7b3e49363978430ac3f13ef087870ad0f9a"} Nov 03 07:37:15.815969 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.815951 4409 scope.go:122] "RemoveContainer" containerID="81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a" Nov 03 07:37:15.816270 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:37:15.816240 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a\": container with ID starting with 81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a not found: ID does not exist" containerID="81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a" Nov 03 07:37:15.816372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.816268 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a"} err="failed to get container status \"81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a\": rpc error: code = NotFound desc = could not find container \"81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a\": container with ID starting with 81f0b122d47f48b1170f16e8e80b6c5160bf63e46ee3c642bd6093bd3ced5c1a not found: ID does not exist" Nov 03 07:37:15.816372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:15.816287 4409 scope.go:122] "RemoveContainer" containerID="a5975f9cfe011d0c6dfb5a4b72393dc2f062a4919fcd9d55342fb0e357cccb89" Nov 03 07:37:16.009286 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:16.009197 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf"] Nov 03 07:37:16.013601 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:16.013572 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5bdb10-cb6974847-fmczf"] Nov 03 07:37:16.119956 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:16.119929 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2"] Nov 03 07:37:16.126452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:16.126425 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5bdb10-84c6cb4ffc-89qs2"] Nov 03 07:37:16.914628 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:16.914558 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090c672f-286d-4291-a953-3eddfdadbb15" path="/var/lib/kubelet/pods/090c672f-286d-4291-a953-3eddfdadbb15/volumes" Nov 03 07:37:16.915088 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:16.915070 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad61f40-0c74-4ea0-9dbe-1361ea562cba" path="/var/lib/kubelet/pods/5ad61f40-0c74-4ea0-9dbe-1361ea562cba/volumes" Nov 03 07:37:18.415959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.415928 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb"] Nov 03 07:37:18.416539 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.416525 4409 cpu_manager.go:414] "containerMap: removing container" podUID="090c672f-286d-4291-a953-3eddfdadbb15" containerName="payload-pre-processing" Nov 03 07:37:18.416578 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.416540 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090c672f-286d-4291-a953-3eddfdadbb15" containerName="payload-pre-processing" Nov 03 07:37:18.416578 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.416575 4409 cpu_manager.go:414] "containerMap: removing container" podUID="5ad61f40-0c74-4ea0-9dbe-1361ea562cba" containerName="payload-processing" Nov 03 07:37:18.416655 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.416581 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad61f40-0c74-4ea0-9dbe-1361ea562cba" containerName="payload-processing" Nov 03 07:37:18.416691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.416673 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad61f40-0c74-4ea0-9dbe-1361ea562cba" containerName="payload-processing" Nov 03 07:37:18.416691 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.416688 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="090c672f-286d-4291-a953-3eddfdadbb15" containerName="payload-pre-processing" Nov 03 07:37:18.421693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.421671 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.475188 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.475157 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-envoy\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.475361 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.475195 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-socket\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.475452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.475360 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-credential-socket\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.475452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.475395 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25689a41-3fdd-401f-a99c-d59679618916-istio-podinfo\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.475452 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.475413 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25689a41-3fdd-401f-a99c-d59679618916-istiod-ca-cert\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.475602 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.475571 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-certs\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.475673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.475613 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rn4g\" (UniqueName: \"kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-kube-api-access-9rn4g\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.475673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.475649 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-data\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.475812 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.475731 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-istio-token\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.499661 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.499626 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb"] Nov 03 07:37:18.527480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.527452 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1fb4bc-openshift-default-dockercfg-pq25s\"" Nov 03 07:37:18.589866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.589828 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-credential-socket\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.590231 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.590193 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25689a41-3fdd-401f-a99c-d59679618916-istio-podinfo\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.590422 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.590400 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25689a41-3fdd-401f-a99c-d59679618916-istiod-ca-cert\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.590679 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.590659 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-certs\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.590887 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.590863 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-credential-socket\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.590998 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.590879 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rn4g\" (UniqueName: \"kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-kube-api-access-9rn4g\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.591132 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.591116 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-data\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.591365 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.591333 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-istio-token\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.591783 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.591763 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-envoy\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.592031 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.592013 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-socket\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.593146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.593092 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-socket\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.593881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.593830 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-certs\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.595006 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.594986 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-data\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.595598 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.591491 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25689a41-3fdd-401f-a99c-d59679618916-istiod-ca-cert\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.599522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.599489 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25689a41-3fdd-401f-a99c-d59679618916-istio-podinfo\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.646344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.646309 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-envoy\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.660066 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.660031 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-istio-token\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.660257 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.660235 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rn4g\" (UniqueName: \"kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-kube-api-access-9rn4g\") pod \"e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:18.734891 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:18.734810 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:19.189852 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:19.189807 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb"] Nov 03 07:37:19.235591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:19.235546 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:19.235977 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:19.235952 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:19.236343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:19.236229 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:19.926506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:19.925419 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" event={"ID":"25689a41-3fdd-401f-a99c-d59679618916","Type":"ContainerStarted","Data":"9cfec25f7888e1c10dbad56265d7919ef0c2179bdadcf2f4ee02a531777cdb65"} Nov 03 07:37:19.926506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:19.925486 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" event={"ID":"25689a41-3fdd-401f-a99c-d59679618916","Type":"ContainerStarted","Data":"1ba8f88ec2cbc454c5e1aa48e60878c7c2fbc51fdeefd8b70c35fd74dd33408f"} Nov 03 07:37:20.067737 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.062698 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74"] Nov 03 07:37:20.082734 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.075443 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" podStartSLOduration=2.075423637 podStartE2EDuration="2.075423637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:20.072031347 +0000 UTC m=+2295.264573098" watchObservedRunningTime="2026-11-03 07:37:20.075423637 +0000 UTC m=+2295.267965387" Nov 03 07:37:20.122934 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.122898 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" Nov 03 07:37:20.157270 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.141081 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74"] Nov 03 07:37:20.204680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.204651 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a8983d-dockercfg-gd7wd\"" Nov 03 07:37:20.206218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.205529 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a8983d\"" Nov 03 07:37:20.325140 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.325082 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43579285-f289-4bdc-8e96-94c3798f3de1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74\" (UID: \"43579285-f289-4bdc-8e96-94c3798f3de1\") " pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" Nov 03 07:37:20.325468 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.325445 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhmnt\" (UniqueName: \"kubernetes.io/projected/43579285-f289-4bdc-8e96-94c3798f3de1-kube-api-access-bhmnt\") pod \"payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74\" (UID: \"43579285-f289-4bdc-8e96-94c3798f3de1\") " pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" Nov 03 07:37:20.329470 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.329449 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv"] Nov 03 07:37:20.337920 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.337895 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" Nov 03 07:37:20.349962 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.349933 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv"] Nov 03 07:37:20.426659 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.426618 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43579285-f289-4bdc-8e96-94c3798f3de1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74\" (UID: \"43579285-f289-4bdc-8e96-94c3798f3de1\") " pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" Nov 03 07:37:20.426871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.426724 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhmnt\" (UniqueName: \"kubernetes.io/projected/43579285-f289-4bdc-8e96-94c3798f3de1-kube-api-access-bhmnt\") pod \"payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74\" (UID: \"43579285-f289-4bdc-8e96-94c3798f3de1\") " pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" Nov 03 07:37:20.427309 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.427289 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43579285-f289-4bdc-8e96-94c3798f3de1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74\" (UID: \"43579285-f289-4bdc-8e96-94c3798f3de1\") " pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" Nov 03 07:37:20.493223 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.493157 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhmnt\" (UniqueName: \"kubernetes.io/projected/43579285-f289-4bdc-8e96-94c3798f3de1-kube-api-access-bhmnt\") pod \"payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74\" (UID: \"43579285-f289-4bdc-8e96-94c3798f3de1\") " pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" Nov 03 07:37:20.536228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.536184 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a5d08c-8abf-4e6f-8977-2485f249882c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv\" (UID: \"93a5d08c-8abf-4e6f-8977-2485f249882c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" Nov 03 07:37:20.537054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.537023 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf8vg\" (UniqueName: \"kubernetes.io/projected/93a5d08c-8abf-4e6f-8977-2485f249882c-kube-api-access-gf8vg\") pod \"payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv\" (UID: \"93a5d08c-8abf-4e6f-8977-2485f249882c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" Nov 03 07:37:20.538186 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.537651 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" Nov 03 07:37:20.641564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.640354 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a5d08c-8abf-4e6f-8977-2485f249882c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv\" (UID: \"93a5d08c-8abf-4e6f-8977-2485f249882c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" Nov 03 07:37:20.641564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.640542 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf8vg\" (UniqueName: \"kubernetes.io/projected/93a5d08c-8abf-4e6f-8977-2485f249882c-kube-api-access-gf8vg\") pod \"payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv\" (UID: \"93a5d08c-8abf-4e6f-8977-2485f249882c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" Nov 03 07:37:20.642812 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.642762 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a5d08c-8abf-4e6f-8977-2485f249882c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv\" (UID: \"93a5d08c-8abf-4e6f-8977-2485f249882c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" Nov 03 07:37:20.656924 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.656898 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f"] Nov 03 07:37:20.663279 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.663261 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.733735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.732951 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-50ae96-openshift-default-dockercfg-qm8zn\"" Nov 03 07:37:20.757851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.739387 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:20.773740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.764771 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-certs\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.773740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.764850 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rglcf\" (UniqueName: \"kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-kube-api-access-rglcf\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.773740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.764901 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-envoy\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.773740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.764952 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-credential-socket\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.788809 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.785259 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f"] Nov 03 07:37:20.856746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.811441 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-socket\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.856746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.811802 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-data\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.856746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.811864 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-podinfo\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.856746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.811909 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-token\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.856746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.811971 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istiod-ca-cert\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.941015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.938766 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-envoy\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.941015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.938840 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-credential-socket\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.941015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.938972 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-socket\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.941015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.939077 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-data\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.941015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.939124 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-podinfo\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.941015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.939163 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-token\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.941015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.939213 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istiod-ca-cert\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.941015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.939315 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-certs\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.941015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.939390 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rglcf\" (UniqueName: \"kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-kube-api-access-rglcf\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:20.946515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.944750 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf8vg\" (UniqueName: \"kubernetes.io/projected/93a5d08c-8abf-4e6f-8977-2485f249882c-kube-api-access-gf8vg\") pod \"payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv\" (UID: \"93a5d08c-8abf-4e6f-8977-2485f249882c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" Nov 03 07:37:21.016823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.956602 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-data\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:21.016823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:20.957428 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-credential-socket\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:21.016823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.007539 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-socket\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:21.016823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.008925 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istiod-ca-cert\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:21.016823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.009521 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" Nov 03 07:37:21.027324 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.027295 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-envoy\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:21.029889 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.029869 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-certs\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:21.033757 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.033671 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rglcf\" (UniqueName: \"kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-kube-api-access-rglcf\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:21.035148 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.035127 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:21.035288 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.035262 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-token\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:21.035409 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.035276 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-podinfo\") pod \"e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:21.035524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.035507 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:21.035638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.035623 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74"] Nov 03 07:37:21.035829 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.035815 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:37:21.062596 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.062572 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:21.464024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.463997 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv"] Nov 03 07:37:21.464587 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:37:21.464555 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a5d08c_8abf_4e6f_8977_2485f249882c.slice/crio-60c3541ff9aa5bcf726e7404498968a29bd639b8f15fa35855da43ee113020e8 WatchSource:0}: Error finding container 60c3541ff9aa5bcf726e7404498968a29bd639b8f15fa35855da43ee113020e8: Status 404 returned error can't find the container with id 60c3541ff9aa5bcf726e7404498968a29bd639b8f15fa35855da43ee113020e8 Nov 03 07:37:21.517531 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.517504 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f"] Nov 03 07:37:21.519031 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:37:21.519005 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7591fd6e_392a_4aa0_8b3f_2ba98cc00db4.slice/crio-87bbcc5f9d8884b50b9abdfc728628df99f2341a1c1e3d9f8b60f6541baf28dc WatchSource:0}: Error finding container 87bbcc5f9d8884b50b9abdfc728628df99f2341a1c1e3d9f8b60f6541baf28dc: Status 404 returned error can't find the container with id 87bbcc5f9d8884b50b9abdfc728628df99f2341a1c1e3d9f8b60f6541baf28dc Nov 03 07:37:21.522333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.522285 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:21.522384 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.522348 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:21.522384 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:21.522380 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:22.015158 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.015123 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" event={"ID":"43579285-f289-4bdc-8e96-94c3798f3de1","Type":"ContainerStarted","Data":"79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e"} Nov 03 07:37:22.015627 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.015181 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" event={"ID":"43579285-f289-4bdc-8e96-94c3798f3de1","Type":"ContainerStarted","Data":"0ab6e51549209b7faee540325f077ec3da0245ec77a758b78aac410e8ca701e5"} Nov 03 07:37:22.015627 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.015253 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" Nov 03 07:37:22.017266 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.017234 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" event={"ID":"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4","Type":"ContainerStarted","Data":"8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa"} Nov 03 07:37:22.017409 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.017279 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" event={"ID":"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4","Type":"ContainerStarted","Data":"87bbcc5f9d8884b50b9abdfc728628df99f2341a1c1e3d9f8b60f6541baf28dc"} Nov 03 07:37:22.018979 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.018956 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" event={"ID":"93a5d08c-8abf-4e6f-8977-2485f249882c","Type":"ContainerStarted","Data":"842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd"} Nov 03 07:37:22.019080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.018994 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" event={"ID":"93a5d08c-8abf-4e6f-8977-2485f249882c","Type":"ContainerStarted","Data":"60c3541ff9aa5bcf726e7404498968a29bd639b8f15fa35855da43ee113020e8"} Nov 03 07:37:22.019080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.019028 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" Nov 03 07:37:22.063621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.063602 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:22.095060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.095033 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:22.140762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.140668 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" podStartSLOduration=3.140606246 podStartE2EDuration="3.140606246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:22.140051505 +0000 UTC m=+2297.332593255" watchObservedRunningTime="2026-11-03 07:37:22.140606246 +0000 UTC m=+2297.333147997" Nov 03 07:37:22.276798 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.276306 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" podStartSLOduration=2.276267921 podStartE2EDuration="2.276267921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:22.269192709 +0000 UTC m=+2297.461734461" watchObservedRunningTime="2026-11-03 07:37:22.276267921 +0000 UTC m=+2297.468809672" Nov 03 07:37:22.394738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:22.389843 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" podStartSLOduration=2.389823422 podStartE2EDuration="2.389823422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:22.388106933 +0000 UTC m=+2297.580648683" watchObservedRunningTime="2026-11-03 07:37:22.389823422 +0000 UTC m=+2297.582365167" Nov 03 07:37:23.025344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:23.025305 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:23.026033 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:23.026008 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:37:29.067385 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:29.067355 4409 generic.go:356] "Generic (PLEG): container finished" podID="d816bfac-359f-463e-81b9-accd32ce79e1" containerID="bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11" exitCode=0 Nov 03 07:37:29.067874 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:29.067441 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" event={"ID":"d816bfac-359f-463e-81b9-accd32ce79e1","Type":"ContainerDied","Data":"bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11"} Nov 03 07:37:29.067977 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:29.067959 4409 scope.go:122] "RemoveContainer" containerID="bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11" Nov 03 07:37:29.069772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:29.069738 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:29.069875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:29.069835 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:29.069919 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:29.069883 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:30.097842 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:30.097806 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" event={"ID":"d816bfac-359f-463e-81b9-accd32ce79e1","Type":"ContainerStarted","Data":"e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b"} Nov 03 07:37:30.600896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:30.600859 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:37:30.602229 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:30.602203 4409 patch_prober.go:29] interesting pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 07:37:30.602369 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:30.602248 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 07:37:31.601002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:31.600960 4409 patch_prober.go:29] interesting pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 07:37:31.601425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:31.601018 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 07:37:32.328581 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.328550 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk"] Nov 03 07:37:32.332786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.332763 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" Nov 03 07:37:32.397306 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.397276 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk"] Nov 03 07:37:32.450796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.450658 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-50ae96\"" Nov 03 07:37:32.451307 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.451285 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-50ae96-dockercfg-7mnpm\"" Nov 03 07:37:32.547000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.546791 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch2pc\" (UniqueName: \"kubernetes.io/projected/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-kube-api-access-ch2pc\") pod \"payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk\" (UID: \"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" Nov 03 07:37:32.547415 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.547389 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk\" (UID: \"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" Nov 03 07:37:32.626017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.625200 4409 patch_prober.go:29] interesting pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 07:37:32.626017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.625267 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 07:37:32.652649 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.652598 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch2pc\" (UniqueName: \"kubernetes.io/projected/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-kube-api-access-ch2pc\") pod \"payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk\" (UID: \"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" Nov 03 07:37:32.652871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.652794 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk\" (UID: \"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" Nov 03 07:37:32.653700 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.653670 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk\" (UID: \"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" Nov 03 07:37:32.716237 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.716208 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch2pc\" (UniqueName: \"kubernetes.io/projected/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-kube-api-access-ch2pc\") pod \"payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk\" (UID: \"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" Nov 03 07:37:32.948053 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:32.947971 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" Nov 03 07:37:33.030549 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:33.030343 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" Nov 03 07:37:33.030549 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:33.030451 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" Nov 03 07:37:33.362479 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:33.362448 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk"] Nov 03 07:37:33.363784 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:37:33.363753 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecf4c8c_f44c_4cb7_9c7a_2a76f6d5a456.slice/crio-59adeb4f5909379f609735be9c5746525b55078ee9ad800851168536878ddfea WatchSource:0}: Error finding container 59adeb4f5909379f609735be9c5746525b55078ee9ad800851168536878ddfea: Status 404 returned error can't find the container with id 59adeb4f5909379f609735be9c5746525b55078ee9ad800851168536878ddfea Nov 03 07:37:33.549404 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:33.549378 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:37:33.600752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:33.600703 4409 patch_prober.go:29] interesting pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 07:37:33.600923 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:33.600772 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 07:37:34.211364 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:34.211329 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" event={"ID":"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456","Type":"ContainerStarted","Data":"d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332"} Nov 03 07:37:34.211913 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:34.211389 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" event={"ID":"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456","Type":"ContainerStarted","Data":"59adeb4f5909379f609735be9c5746525b55078ee9ad800851168536878ddfea"} Nov 03 07:37:34.211913 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:34.211421 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" Nov 03 07:37:34.332556 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:34.332502 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" podStartSLOduration=2.332489117 podStartE2EDuration="2.332489117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:34.328936632 +0000 UTC m=+2309.521478385" watchObservedRunningTime="2026-11-03 07:37:34.332489117 +0000 UTC m=+2309.525030882" Nov 03 07:37:34.601317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:34.601278 4409 patch_prober.go:29] interesting pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 07:37:34.601525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:34.601333 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 07:37:35.606189 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:35.606160 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:37:35.607401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:35.607376 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:37:38.775702 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:37:38.775663 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:37:39.621668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:39.621632 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r"] Nov 03 07:37:39.626337 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:39.626309 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" Nov 03 07:37:39.683968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:39.683935 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r"] Nov 03 07:37:39.742739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:39.731329 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-dcdb23\"" Nov 03 07:37:39.743142 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:39.743116 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-dcdb23-dockercfg-gsm25\"" Nov 03 07:37:39.819927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:39.819893 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgrhn\" (UniqueName: \"kubernetes.io/projected/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-kube-api-access-lgrhn\") pod \"payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r\" (UID: \"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" Nov 03 07:37:39.820655 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:39.820597 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r\" (UID: \"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" Nov 03 07:37:39.936781 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:39.936677 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgrhn\" (UniqueName: \"kubernetes.io/projected/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-kube-api-access-lgrhn\") pod \"payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r\" (UID: \"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" Nov 03 07:37:39.936781 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:39.936771 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r\" (UID: \"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" Nov 03 07:37:39.937535 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:39.937506 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r\" (UID: \"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" Nov 03 07:37:39.999890 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:39.999853 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgrhn\" (UniqueName: \"kubernetes.io/projected/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-kube-api-access-lgrhn\") pod \"payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r\" (UID: \"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" Nov 03 07:37:40.240498 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:40.240396 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" Nov 03 07:37:40.622724 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:40.622678 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r"] Nov 03 07:37:40.624210 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:37:40.624179 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c50c34_72c7_4e1b_a926_4e9ef47b9fd5.slice/crio-cd0da443f9bf029b3cc5bc2ff57bd23cb2e7529cdc31f6b83f13876bc343f866 WatchSource:0}: Error finding container cd0da443f9bf029b3cc5bc2ff57bd23cb2e7529cdc31f6b83f13876bc343f866: Status 404 returned error can't find the container with id cd0da443f9bf029b3cc5bc2ff57bd23cb2e7529cdc31f6b83f13876bc343f866 Nov 03 07:37:41.260288 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:41.260253 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" event={"ID":"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5","Type":"ContainerStarted","Data":"050790d2157c24854bc0d57c2e21ba969f3f8df015ddd4296a99874beaec7ea7"} Nov 03 07:37:41.260772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:41.260305 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" event={"ID":"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5","Type":"ContainerStarted","Data":"cd0da443f9bf029b3cc5bc2ff57bd23cb2e7529cdc31f6b83f13876bc343f866"} Nov 03 07:37:41.260772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:41.260374 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" Nov 03 07:37:41.380059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:41.379993 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" podStartSLOduration=2.379977425 podStartE2EDuration="2.379977425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:41.377147636 +0000 UTC m=+2316.569689387" watchObservedRunningTime="2026-11-03 07:37:41.379977425 +0000 UTC m=+2316.572519172" Nov 03 07:37:45.220522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:45.220494 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" Nov 03 07:37:52.106132 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.106090 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r"] Nov 03 07:37:52.110744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.110692 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.155666 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.155641 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw"] Nov 03 07:37:52.159788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.159773 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.203444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.203416 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.203569 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.203514 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.203699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.203679 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.203795 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.203744 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.203859 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.203828 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.203909 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.203878 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.203959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.203937 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.204000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.203982 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.204042 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.204011 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwn8l\" (UniqueName: \"kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-kube-api-access-dwn8l\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.259663 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.259629 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw"] Nov 03 07:37:52.310767 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.307653 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.311105 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.311076 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.311277 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.311262 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.311409 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.311395 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.311676 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.311522 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.311960 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.311930 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.312128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.312110 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.312261 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.312247 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.312408 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.312392 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.312552 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.312535 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwn8l\" (UniqueName: \"kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-kube-api-access-dwn8l\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.312732 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.312695 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.312862 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.312845 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.313028 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.313014 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.313161 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.313145 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.313375 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.313361 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7gc\" (UniqueName: \"kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-kube-api-access-ch7gc\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.313529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.313513 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.313639 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.313626 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.313823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.313799 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.314503 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.314479 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.315345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.315320 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.316451 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.316425 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.316630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.316611 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" Nov 03 07:37:52.317191 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.317159 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.318267 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.317484 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.320544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.320479 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.322534 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.322508 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.338640 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.338611 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r"] Nov 03 07:37:52.378064 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.377917 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.379635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.379611 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwn8l\" (UniqueName: \"kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-kube-api-access-dwn8l\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.414756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.414697 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.414873 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.414792 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.414873 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.414871 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7gc\" (UniqueName: \"kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-kube-api-access-ch7gc\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.415007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.414904 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.415007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.414928 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.415007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.414993 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.415173 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.415072 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.415173 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.415157 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.415301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.415218 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.415301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.415232 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-certs\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.415407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.415296 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.415462 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.415424 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-credential-socket\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.415577 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.415558 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-data\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.416119 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.416083 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istiod-ca-cert\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.417068 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.417051 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-envoy\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.417539 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.417520 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-podinfo\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.424704 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.424683 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:52.477879 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.477848 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-token\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.478036 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.477895 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7gc\" (UniqueName: \"kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-kube-api-access-ch7gc\") pod \"e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.772868 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.772791 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:52.863048 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.863015 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r"] Nov 03 07:37:52.866121 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:37:52.866095 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f8a61a_8d5d_43b8_b266_cdc49da02fb1.slice/crio-9696dd9711a35bf781ffa74575ccdf4e31160cc63b76e566eef59f412481a74a WatchSource:0}: Error finding container 9696dd9711a35bf781ffa74575ccdf4e31160cc63b76e566eef59f412481a74a: Status 404 returned error can't find the container with id 9696dd9711a35bf781ffa74575ccdf4e31160cc63b76e566eef59f412481a74a Nov 03 07:37:52.869482 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.869449 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:52.869581 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.869524 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:52.869581 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:52.869555 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:53.162813 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.162790 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw"] Nov 03 07:37:53.163335 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:37:53.163312 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13f69f0_6a0a_4278_aedd_e2248f2f2cb4.slice/crio-3aabb0fac1bb59b42e6db89be46d4f77aa3300f4906bc5aba35aba64fa210c0a WatchSource:0}: Error finding container 3aabb0fac1bb59b42e6db89be46d4f77aa3300f4906bc5aba35aba64fa210c0a: Status 404 returned error can't find the container with id 3aabb0fac1bb59b42e6db89be46d4f77aa3300f4906bc5aba35aba64fa210c0a Nov 03 07:37:53.166301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.166260 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:53.166375 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.166346 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:53.166418 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.166387 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:37:53.362628 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.361399 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" event={"ID":"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4","Type":"ContainerStarted","Data":"5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434"} Nov 03 07:37:53.362628 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.361470 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" event={"ID":"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4","Type":"ContainerStarted","Data":"3aabb0fac1bb59b42e6db89be46d4f77aa3300f4906bc5aba35aba64fa210c0a"} Nov 03 07:37:53.365204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.365175 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" event={"ID":"08f8a61a-8d5d-43b8-b266-cdc49da02fb1","Type":"ContainerStarted","Data":"d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79"} Nov 03 07:37:53.365355 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.365223 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" event={"ID":"08f8a61a-8d5d-43b8-b266-cdc49da02fb1","Type":"ContainerStarted","Data":"9696dd9711a35bf781ffa74575ccdf4e31160cc63b76e566eef59f412481a74a"} Nov 03 07:37:53.425574 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.425497 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:53.427050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.427025 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 03 07:37:53.427195 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.427081 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" podUID="08f8a61a-8d5d-43b8-b266-cdc49da02fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 03 07:37:53.500330 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.500276 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" podStartSLOduration=1.500260381 podStartE2EDuration="1.500260381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:53.494320256 +0000 UTC m=+2328.686862011" watchObservedRunningTime="2026-11-03 07:37:53.500260381 +0000 UTC m=+2328.692802128" Nov 03 07:37:53.617772 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.617688 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" podStartSLOduration=1.617673979 podStartE2EDuration="1.617673979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:37:53.615743893 +0000 UTC m=+2328.808285643" watchObservedRunningTime="2026-11-03 07:37:53.617673979 +0000 UTC m=+2328.810215728" Nov 03 07:37:53.784793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:53.776802 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:54.425429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:54.425257 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 03 07:37:54.425429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:54.425309 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" podUID="08f8a61a-8d5d-43b8-b266-cdc49da02fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 03 07:37:54.777798 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:54.774754 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 03 07:37:54.777798 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:54.774817 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" podUID="c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 03 07:37:55.425138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:55.425103 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 03 07:37:55.425312 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:55.425151 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" podUID="08f8a61a-8d5d-43b8-b266-cdc49da02fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 03 07:37:55.791804 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:55.788050 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 03 07:37:55.791804 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:55.788109 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" podUID="c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 03 07:37:56.425127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:56.425079 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 03 07:37:56.425339 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:56.425146 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" podUID="08f8a61a-8d5d-43b8-b266-cdc49da02fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 03 07:37:56.775204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:56.775115 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 03 07:37:56.775463 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:56.775432 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" podUID="c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 03 07:37:56.924868 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:56.924842 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr"] Nov 03 07:37:56.925567 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:56.925537 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" podUID="a6938ea1-064f-40fe-8e6e-670a4a9615f4" containerName="maas-api" containerID="cri-o://0d38589c4ae94ff76d0b1b3039bb5fce031197a3501b0030bbea4a6e70d10aaa" gracePeriod=30 Nov 03 07:37:57.433764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.425113 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 03 07:37:57.433764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.425177 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" podUID="08f8a61a-8d5d-43b8-b266-cdc49da02fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 03 07:37:57.553026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.552959 4409 generic.go:356] "Generic (PLEG): container finished" podID="a6938ea1-064f-40fe-8e6e-670a4a9615f4" containerID="0d38589c4ae94ff76d0b1b3039bb5fce031197a3501b0030bbea4a6e70d10aaa" exitCode=0 Nov 03 07:37:57.553026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.552992 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" event={"ID":"a6938ea1-064f-40fe-8e6e-670a4a9615f4","Type":"ContainerDied","Data":"0d38589c4ae94ff76d0b1b3039bb5fce031197a3501b0030bbea4a6e70d10aaa"} Nov 03 07:37:57.631259 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.630944 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:37:57.716181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.716069 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-metrics-certs\") pod \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " Nov 03 07:37:57.716835 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.716810 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6938ea1-064f-40fe-8e6e-670a4a9615f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6938ea1-064f-40fe-8e6e-670a4a9615f4-openshift-service-ca\") pod \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " Nov 03 07:37:57.717577 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.717399 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6938ea1-064f-40fe-8e6e-670a4a9615f4-openshift-service-ca" pod "a6938ea1-064f-40fe-8e6e-670a4a9615f4" (UID: "a6938ea1-064f-40fe-8e6e-670a4a9615f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:37:57.717793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.717771 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-maas-api-tls\") pod \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " Nov 03 07:37:57.718229 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.718216 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6938ea1-064f-40fe-8e6e-670a4a9615f4-kube-api-access-t784n\" (UniqueName: \"kubernetes.io/projected/a6938ea1-064f-40fe-8e6e-670a4a9615f4-kube-api-access-t784n\") pod \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\" (UID: \"a6938ea1-064f-40fe-8e6e-670a4a9615f4\") " Nov 03 07:37:57.719126 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.719112 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6938ea1-064f-40fe-8e6e-670a4a9615f4-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:57.719963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.719948 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-maas-api-tls" pod "a6938ea1-064f-40fe-8e6e-670a4a9615f4" (UID: "a6938ea1-064f-40fe-8e6e-670a4a9615f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:37:57.720061 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.720035 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-metrics-certs" pod "a6938ea1-064f-40fe-8e6e-670a4a9615f4" (UID: "a6938ea1-064f-40fe-8e6e-670a4a9615f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:37:57.720531 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.720512 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6938ea1-064f-40fe-8e6e-670a4a9615f4-kube-api-access-t784n" pod "a6938ea1-064f-40fe-8e6e-670a4a9615f4" (UID: "a6938ea1-064f-40fe-8e6e-670a4a9615f4"). InnerVolumeSpecName "kube-api-access-t784n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:37:57.795950 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.793325 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 03 07:37:57.795950 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.793388 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" podUID="c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 03 07:37:57.824921 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.824879 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:57.824921 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.824908 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6938ea1-064f-40fe-8e6e-670a4a9615f4-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:57.824921 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:57.824919 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t784n\" (UniqueName: \"kubernetes.io/projected/a6938ea1-064f-40fe-8e6e-670a4a9615f4-kube-api-access-t784n\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:37:58.426243 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:58.426208 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 03 07:37:58.426637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:58.426268 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" podUID="08f8a61a-8d5d-43b8-b266-cdc49da02fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 03 07:37:58.561500 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:58.561468 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" event={"ID":"a6938ea1-064f-40fe-8e6e-670a4a9615f4","Type":"ContainerDied","Data":"39f5cb8b02451cb3da870bff7ac763834ae4b1404225ba8fcf8a8276e32386c6"} Nov 03 07:37:58.561500 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:58.561493 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" Nov 03 07:37:58.561702 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:58.561525 4409 scope.go:122] "RemoveContainer" containerID="0d38589c4ae94ff76d0b1b3039bb5fce031197a3501b0030bbea4a6e70d10aaa" Nov 03 07:37:58.748130 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:58.748106 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr"] Nov 03 07:37:58.777177 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:58.777136 4409 patch_prober.go:29] interesting pod/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 03 07:37:58.785836 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:58.785784 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" podUID="c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 03 07:37:58.821259 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:58.821228 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr"] Nov 03 07:37:58.920736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:58.920675 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6938ea1-064f-40fe-8e6e-670a4a9615f4" path="/var/lib/kubelet/pods/a6938ea1-064f-40fe-8e6e-670a4a9615f4/volumes" Nov 03 07:37:59.608151 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:59.608115 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:59.608774 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:59.608754 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:59.609808 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:59.609774 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:37:59.908529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:59.908370 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:59.912744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:59.911557 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:37:59.912922 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:37:59.912854 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:38:00.480791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:00.480755 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5bdb10-84fbf64c9b-8wmrr" podUID="a6938ea1-064f-40fe-8e6e-670a4a9615f4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.120:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:38:00.578402 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:00.578375 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm"] Nov 03 07:38:00.689199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:00.689164 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v"] Nov 03 07:38:05.579403 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:05.579361 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" podUID="c8b8b2c6-1104-477c-af9c-5fe063f450c5" containerName="payload-processing" containerID="cri-o://c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0" gracePeriod=25 Nov 03 07:38:05.690258 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:05.690199 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" podUID="5714b44e-b0be-419d-8106-94e5a28f9e55" containerName="payload-pre-processing" containerID="cri-o://e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e" gracePeriod=25 Nov 03 07:38:05.880964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:05.880934 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" Nov 03 07:38:05.993483 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:05.993459 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" Nov 03 07:38:06.050925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.050880 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b8b2c6-1104-477c-af9c-5fe063f450c5-kube-api-access-rkppp\" (UniqueName: \"kubernetes.io/projected/c8b8b2c6-1104-477c-af9c-5fe063f450c5-kube-api-access-rkppp\") pod \"c8b8b2c6-1104-477c-af9c-5fe063f450c5\" (UID: \"c8b8b2c6-1104-477c-af9c-5fe063f450c5\") " Nov 03 07:38:06.051102 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.051055 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b8b2c6-1104-477c-af9c-5fe063f450c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b8b2c6-1104-477c-af9c-5fe063f450c5-plugins-config-volume\") pod \"c8b8b2c6-1104-477c-af9c-5fe063f450c5\" (UID: \"c8b8b2c6-1104-477c-af9c-5fe063f450c5\") " Nov 03 07:38:06.051420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.051396 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b8b2c6-1104-477c-af9c-5fe063f450c5-plugins-config-volume" pod "c8b8b2c6-1104-477c-af9c-5fe063f450c5" (UID: "c8b8b2c6-1104-477c-af9c-5fe063f450c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:38:06.051682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.051667 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b8b2c6-1104-477c-af9c-5fe063f450c5-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:06.053165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.053146 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b8b2c6-1104-477c-af9c-5fe063f450c5-kube-api-access-rkppp" pod "c8b8b2c6-1104-477c-af9c-5fe063f450c5" (UID: "c8b8b2c6-1104-477c-af9c-5fe063f450c5"). InnerVolumeSpecName "kube-api-access-rkppp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:38:06.153037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.152956 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5714b44e-b0be-419d-8106-94e5a28f9e55-kube-api-access-pqzc9\" (UniqueName: \"kubernetes.io/projected/5714b44e-b0be-419d-8106-94e5a28f9e55-kube-api-access-pqzc9\") pod \"5714b44e-b0be-419d-8106-94e5a28f9e55\" (UID: \"5714b44e-b0be-419d-8106-94e5a28f9e55\") " Nov 03 07:38:06.153370 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.153348 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5714b44e-b0be-419d-8106-94e5a28f9e55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5714b44e-b0be-419d-8106-94e5a28f9e55-plugins-config-volume\") pod \"5714b44e-b0be-419d-8106-94e5a28f9e55\" (UID: \"5714b44e-b0be-419d-8106-94e5a28f9e55\") " Nov 03 07:38:06.153706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.153679 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5714b44e-b0be-419d-8106-94e5a28f9e55-plugins-config-volume" pod "5714b44e-b0be-419d-8106-94e5a28f9e55" (UID: "5714b44e-b0be-419d-8106-94e5a28f9e55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:38:06.154081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.154061 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkppp\" (UniqueName: \"kubernetes.io/projected/c8b8b2c6-1104-477c-af9c-5fe063f450c5-kube-api-access-rkppp\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:06.154081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.154082 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5714b44e-b0be-419d-8106-94e5a28f9e55-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:06.155763 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.155700 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5714b44e-b0be-419d-8106-94e5a28f9e55-kube-api-access-pqzc9" pod "5714b44e-b0be-419d-8106-94e5a28f9e55" (UID: "5714b44e-b0be-419d-8106-94e5a28f9e55"). InnerVolumeSpecName "kube-api-access-pqzc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:38:06.255621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.255588 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqzc9\" (UniqueName: \"kubernetes.io/projected/5714b44e-b0be-419d-8106-94e5a28f9e55-kube-api-access-pqzc9\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:06.634447 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.634411 4409 generic.go:356] "Generic (PLEG): container finished" podID="c8b8b2c6-1104-477c-af9c-5fe063f450c5" containerID="c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0" exitCode=0 Nov 03 07:38:06.634936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.634459 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" Nov 03 07:38:06.634936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.634498 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" event={"ID":"c8b8b2c6-1104-477c-af9c-5fe063f450c5","Type":"ContainerDied","Data":"c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0"} Nov 03 07:38:06.634936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.634565 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm" event={"ID":"c8b8b2c6-1104-477c-af9c-5fe063f450c5","Type":"ContainerDied","Data":"cddd9d69b1c2c80d6c6294e8f48a7b626c02c41264bcf15581425487de72cbba"} Nov 03 07:38:06.634936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.634600 4409 scope.go:122] "RemoveContainer" containerID="c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0" Nov 03 07:38:06.636183 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.636163 4409 generic.go:356] "Generic (PLEG): container finished" podID="5714b44e-b0be-419d-8106-94e5a28f9e55" containerID="e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e" exitCode=0 Nov 03 07:38:06.636308 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.636285 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" event={"ID":"5714b44e-b0be-419d-8106-94e5a28f9e55","Type":"ContainerDied","Data":"e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e"} Nov 03 07:38:06.636355 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.636331 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" event={"ID":"5714b44e-b0be-419d-8106-94e5a28f9e55","Type":"ContainerDied","Data":"606f8c38733cadd8e3e68a94de82d9e1d276da8673988d4f2fb2bda6db3952c7"} Nov 03 07:38:06.636396 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.636367 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v" Nov 03 07:38:06.647464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.647446 4409 scope.go:122] "RemoveContainer" containerID="c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0" Nov 03 07:38:06.647739 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:38:06.647698 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0\": container with ID starting with c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0 not found: ID does not exist" containerID="c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0" Nov 03 07:38:06.647833 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.647751 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0"} err="failed to get container status \"c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0\": rpc error: code = NotFound desc = could not find container \"c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0\": container with ID starting with c5a168fcd9cc0c03b8b5ec64f0e22f46a9631c3be869a31fbab9a5c1bbbce2b0 not found: ID does not exist" Nov 03 07:38:06.647833 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.647775 4409 scope.go:122] "RemoveContainer" containerID="e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e" Nov 03 07:38:06.657863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.657847 4409 scope.go:122] "RemoveContainer" containerID="e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e" Nov 03 07:38:06.658113 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:38:06.658093 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e\": container with ID starting with e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e not found: ID does not exist" containerID="e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e" Nov 03 07:38:06.658187 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.658121 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e"} err="failed to get container status \"e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e\": rpc error: code = NotFound desc = could not find container \"e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e\": container with ID starting with e0372ecbc1c6e150ffe5e45e3026fff3ee3acb2dcf185f5f1e3551a7a7ba988e not found: ID does not exist" Nov 03 07:38:06.822180 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.808244 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v"] Nov 03 07:38:06.853893 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.853808 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-295fba7c-8d6586c7f-klv4v"] Nov 03 07:38:06.915221 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.915182 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5714b44e-b0be-419d-8106-94e5a28f9e55" path="/var/lib/kubelet/pods/5714b44e-b0be-419d-8106-94e5a28f9e55/volumes" Nov 03 07:38:06.926799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.926769 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm"] Nov 03 07:38:06.934635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:06.934607 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-295fba7c-79db5944d9-fvlzm"] Nov 03 07:38:08.775240 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:38:08.775203 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:38:08.913842 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:08.913803 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b8b2c6-1104-477c-af9c-5fe063f450c5" path="/var/lib/kubelet/pods/c8b8b2c6-1104-477c-af9c-5fe063f450c5/volumes" Nov 03 07:38:25.424321 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.424285 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn"] Nov 03 07:38:25.425157 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.425135 4409 cpu_manager.go:414] "containerMap: removing container" podUID="c8b8b2c6-1104-477c-af9c-5fe063f450c5" containerName="payload-processing" Nov 03 07:38:25.425157 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.425158 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b8b2c6-1104-477c-af9c-5fe063f450c5" containerName="payload-processing" Nov 03 07:38:25.425333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.425193 4409 cpu_manager.go:414] "containerMap: removing container" podUID="a6938ea1-064f-40fe-8e6e-670a4a9615f4" containerName="maas-api" Nov 03 07:38:25.425333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.425202 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6938ea1-064f-40fe-8e6e-670a4a9615f4" containerName="maas-api" Nov 03 07:38:25.425333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.425231 4409 cpu_manager.go:414] "containerMap: removing container" podUID="5714b44e-b0be-419d-8106-94e5a28f9e55" containerName="payload-pre-processing" Nov 03 07:38:25.425333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.425238 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5714b44e-b0be-419d-8106-94e5a28f9e55" containerName="payload-pre-processing" Nov 03 07:38:25.425535 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.425392 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="5714b44e-b0be-419d-8106-94e5a28f9e55" containerName="payload-pre-processing" Nov 03 07:38:25.425535 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.425415 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b8b2c6-1104-477c-af9c-5fe063f450c5" containerName="payload-processing" Nov 03 07:38:25.425535 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.425431 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6938ea1-064f-40fe-8e6e-670a4a9615f4" containerName="maas-api" Nov 03 07:38:25.430278 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.430258 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:25.498779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.498739 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-metrics-certs\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:25.498948 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.498885 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dk27\" (UniqueName: \"kubernetes.io/projected/6a5d1906-d406-4178-8be5-2af12fc35c35-kube-api-access-5dk27\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:25.499043 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.499024 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5d1906-d406-4178-8be5-2af12fc35c35-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:25.499179 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.499070 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-maas-api-tls\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:25.533086 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.533053 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1fb4bc-serving-cert\"" Nov 03 07:38:25.536533 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.536512 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn"] Nov 03 07:38:25.600412 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.600386 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5d1906-d406-4178-8be5-2af12fc35c35-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:25.600577 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.600426 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-maas-api-tls\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:25.600577 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.600502 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-metrics-certs\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:25.600731 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.600579 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dk27\" (UniqueName: \"kubernetes.io/projected/6a5d1906-d406-4178-8be5-2af12fc35c35-kube-api-access-5dk27\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:25.600731 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:38:25.600601 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-serving-cert: secret "maas-api-e2e-api-a-1fb4bc-serving-cert" not found Nov 03 07:38:25.600731 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:38:25.600680 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-maas-api-tls podName:6a5d1906-d406-4178-8be5-2af12fc35c35 nodeName:}" failed. No retries permitted until 2026-11-03 07:38:26.100659314 +0000 UTC m=+2361.293201055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-maas-api-tls") pod "maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" (UID: "6a5d1906-d406-4178-8be5-2af12fc35c35") : secret "maas-api-e2e-api-a-1fb4bc-serving-cert" not found Nov 03 07:38:25.601249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.601220 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5d1906-d406-4178-8be5-2af12fc35c35-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:25.603782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.603757 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-metrics-certs\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:25.694745 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:25.694610 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dk27\" (UniqueName: \"kubernetes.io/projected/6a5d1906-d406-4178-8be5-2af12fc35c35-kube-api-access-5dk27\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:26.108254 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:26.108221 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-maas-api-tls\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:26.110773 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:26.110736 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-maas-api-tls\") pod \"maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:26.344603 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:26.344573 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:26.725982 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:26.725952 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn"] Nov 03 07:38:26.726644 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:38:26.726615 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5d1906_d406_4178_8be5_2af12fc35c35.slice/crio-1630fa4b5a5b9359e7dd9b3769a852fbe1f207a441540a88a56bcd91767b85d1 WatchSource:0}: Error finding container 1630fa4b5a5b9359e7dd9b3769a852fbe1f207a441540a88a56bcd91767b85d1: Status 404 returned error can't find the container with id 1630fa4b5a5b9359e7dd9b3769a852fbe1f207a441540a88a56bcd91767b85d1 Nov 03 07:38:26.821855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:26.821825 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" event={"ID":"6a5d1906-d406-4178-8be5-2af12fc35c35","Type":"ContainerStarted","Data":"1630fa4b5a5b9359e7dd9b3769a852fbe1f207a441540a88a56bcd91767b85d1"} Nov 03 07:38:27.829120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:27.829081 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" event={"ID":"6a5d1906-d406-4178-8be5-2af12fc35c35","Type":"ContainerStarted","Data":"97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618"} Nov 03 07:38:27.829530 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:27.829269 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:27.952786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:27.952730 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" podStartSLOduration=2.136503378 podStartE2EDuration="2.952697641s" totalImagesPullingTime="816.194263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:38:27.950598382 +0000 UTC m=+2363.143140154" watchObservedRunningTime="2026-11-03 07:38:27.952697641 +0000 UTC m=+2363.145239391" Nov 03 07:38:30.432735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.432677 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6"] Nov 03 07:38:30.446105 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.446079 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:30.496900 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.496871 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6"] Nov 03 07:38:30.541627 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.541598 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-50ae96-serving-cert\"" Nov 03 07:38:30.589820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.589782 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-metrics-certs\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:30.590000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.589859 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:30.590350 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.590319 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/471524d1-c544-492a-b04b-4ee01b5bd0e0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:30.590576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.590555 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84sg\" (UniqueName: \"kubernetes.io/projected/471524d1-c544-492a-b04b-4ee01b5bd0e0-kube-api-access-l84sg\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:30.712017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.711937 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-metrics-certs\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:30.719974 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.719942 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:30.726834 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:38:30.720269 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-serving-cert: secret "maas-api-e2e-inf-a-50ae96-serving-cert" not found Nov 03 07:38:30.727148 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:38:30.727129 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-maas-api-tls podName:471524d1-c544-492a-b04b-4ee01b5bd0e0 nodeName:}" failed. No retries permitted until 2026-11-03 07:38:31.227102879 +0000 UTC m=+2366.419644627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-maas-api-tls") pod "maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" (UID: "471524d1-c544-492a-b04b-4ee01b5bd0e0") : secret "maas-api-e2e-inf-a-50ae96-serving-cert" not found Nov 03 07:38:30.727281 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.726908 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/471524d1-c544-492a-b04b-4ee01b5bd0e0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:30.727545 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.727520 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84sg\" (UniqueName: \"kubernetes.io/projected/471524d1-c544-492a-b04b-4ee01b5bd0e0-kube-api-access-l84sg\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:30.727926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.727632 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/471524d1-c544-492a-b04b-4ee01b5bd0e0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:30.733339 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.733263 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-metrics-certs\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:30.792112 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:30.792075 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84sg\" (UniqueName: \"kubernetes.io/projected/471524d1-c544-492a-b04b-4ee01b5bd0e0-kube-api-access-l84sg\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:31.236100 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:31.236070 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:31.238517 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:31.238486 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:31.360215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:31.360182 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:31.362574 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:31.362544 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc"] Nov 03 07:38:31.362768 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:31.362746 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" containerID="cri-o://9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c" gracePeriod=30 Nov 03 07:38:31.758429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:31.758397 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6"] Nov 03 07:38:31.760207 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:38:31.760176 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471524d1_c544_492a_b04b_4ee01b5bd0e0.slice/crio-d3804e35703dd74c9d6b101930d7a82d7b1913ec5f3c009d43ef35a3b30bcdad WatchSource:0}: Error finding container d3804e35703dd74c9d6b101930d7a82d7b1913ec5f3c009d43ef35a3b30bcdad: Status 404 returned error can't find the container with id d3804e35703dd74c9d6b101930d7a82d7b1913ec5f3c009d43ef35a3b30bcdad Nov 03 07:38:31.858287 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:31.858258 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" event={"ID":"471524d1-c544-492a-b04b-4ee01b5bd0e0","Type":"ContainerStarted","Data":"d3804e35703dd74c9d6b101930d7a82d7b1913ec5f3c009d43ef35a3b30bcdad"} Nov 03 07:38:32.865839 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:32.865750 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" event={"ID":"471524d1-c544-492a-b04b-4ee01b5bd0e0","Type":"ContainerStarted","Data":"e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb"} Nov 03 07:38:32.866202 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:32.865940 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:32.986931 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:32.986871 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" podStartSLOduration=2.155474258 podStartE2EDuration="2.986856933s" totalImagesPullingTime="831.382675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:38:32.984266969 +0000 UTC m=+2368.176808716" watchObservedRunningTime="2026-11-03 07:38:32.986856933 +0000 UTC m=+2368.179398683" Nov 03 07:38:33.841071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:33.841044 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:38:36.642079 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.642017 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:38:36.745525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.745491 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-token\" (UniqueName: \"kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-token\") pod \"a951fa97-ef86-4052-b96e-cf304f590ef6\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " Nov 03 07:38:36.745755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.745573 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-data\") pod \"a951fa97-ef86-4052-b96e-cf304f590ef6\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " Nov 03 07:38:36.745755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.745618 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-podinfo\") pod \"a951fa97-ef86-4052-b96e-cf304f590ef6\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " Nov 03 07:38:36.745755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.745699 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a951fa97-ef86-4052-b96e-cf304f590ef6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a951fa97-ef86-4052-b96e-cf304f590ef6-istiod-ca-cert\") pod \"a951fa97-ef86-4052-b96e-cf304f590ef6\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " Nov 03 07:38:36.745972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.745834 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-socket\") pod \"a951fa97-ef86-4052-b96e-cf304f590ef6\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " Nov 03 07:38:36.745972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.745912 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-data" pod "a951fa97-ef86-4052-b96e-cf304f590ef6" (UID: "a951fa97-ef86-4052-b96e-cf304f590ef6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:38:36.745972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.745930 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-certs\") pod \"a951fa97-ef86-4052-b96e-cf304f590ef6\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " Nov 03 07:38:36.746142 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.746001 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-credential-socket\") pod \"a951fa97-ef86-4052-b96e-cf304f590ef6\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " Nov 03 07:38:36.746142 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.746124 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-socket" pod "a951fa97-ef86-4052-b96e-cf304f590ef6" (UID: "a951fa97-ef86-4052-b96e-cf304f590ef6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:38:36.746251 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.746140 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-envoy\") pod \"a951fa97-ef86-4052-b96e-cf304f590ef6\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " Nov 03 07:38:36.746251 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.746190 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-kube-api-access-4bzw7\" (UniqueName: \"kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-kube-api-access-4bzw7\") pod \"a951fa97-ef86-4052-b96e-cf304f590ef6\" (UID: \"a951fa97-ef86-4052-b96e-cf304f590ef6\") " Nov 03 07:38:36.746251 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.746201 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-certs" pod "a951fa97-ef86-4052-b96e-cf304f590ef6" (UID: "a951fa97-ef86-4052-b96e-cf304f590ef6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:38:36.746251 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.746223 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a951fa97-ef86-4052-b96e-cf304f590ef6-istiod-ca-cert" pod "a951fa97-ef86-4052-b96e-cf304f590ef6" (UID: "a951fa97-ef86-4052-b96e-cf304f590ef6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:38:36.746432 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.746352 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-credential-socket" pod "a951fa97-ef86-4052-b96e-cf304f590ef6" (UID: "a951fa97-ef86-4052-b96e-cf304f590ef6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:38:36.746840 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.746816 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.747018 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.746994 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.747018 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.747019 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.747203 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.747033 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.747203 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.747048 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a951fa97-ef86-4052-b96e-cf304f590ef6-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.748657 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.748630 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-token" pod "a951fa97-ef86-4052-b96e-cf304f590ef6" (UID: "a951fa97-ef86-4052-b96e-cf304f590ef6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:38:36.748873 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.748848 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-podinfo" pod "a951fa97-ef86-4052-b96e-cf304f590ef6" (UID: "a951fa97-ef86-4052-b96e-cf304f590ef6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:38:36.748987 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.748962 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-envoy" pod "a951fa97-ef86-4052-b96e-cf304f590ef6" (UID: "a951fa97-ef86-4052-b96e-cf304f590ef6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:38:36.749102 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.749076 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-kube-api-access-4bzw7" pod "a951fa97-ef86-4052-b96e-cf304f590ef6" (UID: "a951fa97-ef86-4052-b96e-cf304f590ef6"). InnerVolumeSpecName "kube-api-access-4bzw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:38:36.848372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.848340 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.848372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.848371 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bzw7\" (UniqueName: \"kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-kube-api-access-4bzw7\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.848572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.848390 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.848572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.848400 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a951fa97-ef86-4052-b96e-cf304f590ef6-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:36.918969 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.918882 4409 generic.go:356] "Generic (PLEG): container finished" podID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerID="9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c" exitCode=0 Nov 03 07:38:36.918969 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.918927 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" Nov 03 07:38:36.919147 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.918968 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" event={"ID":"a951fa97-ef86-4052-b96e-cf304f590ef6","Type":"ContainerDied","Data":"9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c"} Nov 03 07:38:36.919147 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.919022 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc" event={"ID":"a951fa97-ef86-4052-b96e-cf304f590ef6","Type":"ContainerDied","Data":"3f0e7323749e2165318b156b6ec5db06024e4b04662d408b257afab728caa3dc"} Nov 03 07:38:36.919147 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.919046 4409 scope.go:122] "RemoveContainer" containerID="9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c" Nov 03 07:38:36.939278 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.939254 4409 scope.go:122] "RemoveContainer" containerID="9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c" Nov 03 07:38:36.939616 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:38:36.939595 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c\": container with ID starting with 9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c not found: ID does not exist" containerID="9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c" Nov 03 07:38:36.939678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:36.939627 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c"} err="failed to get container status \"9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c\": rpc error: code = NotFound desc = could not find container \"9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c\": container with ID starting with 9e1547f9d336c4368e75c709b2fb99a3b2aa47afa01e14600979fa40b4f7432c not found: ID does not exist" Nov 03 07:38:37.088956 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:37.088925 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc"] Nov 03 07:38:37.095294 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:37.095270 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-7mgjc"] Nov 03 07:38:38.776104 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:38:38.776062 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:38:38.878478 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:38.878453 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:38:38.914617 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:38.914573 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" path="/var/lib/kubelet/pods/a951fa97-ef86-4052-b96e-cf304f590ef6/volumes" Nov 03 07:38:50.196794 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.186975 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6"] Nov 03 07:38:50.213058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.213009 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" podUID="17d55a46-9c7a-4cc7-989a-a27c45ee3e91" containerName="maas-api" containerID="cri-o://2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000" gracePeriod=30 Nov 03 07:38:50.485700 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.485674 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:38:50.671803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.670321 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-maas-api-tls\") pod \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " Nov 03 07:38:50.671803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.670518 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-metrics-certs\") pod \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " Nov 03 07:38:50.671803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.670643 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-kube-api-access-nf9q4\" (UniqueName: \"kubernetes.io/projected/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-kube-api-access-nf9q4\") pod \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " Nov 03 07:38:50.671803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.670683 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-openshift-service-ca\") pod \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\" (UID: \"17d55a46-9c7a-4cc7-989a-a27c45ee3e91\") " Nov 03 07:38:50.673863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.672428 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-openshift-service-ca" pod "17d55a46-9c7a-4cc7-989a-a27c45ee3e91" (UID: "17d55a46-9c7a-4cc7-989a-a27c45ee3e91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:38:50.688140 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.687193 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-kube-api-access-nf9q4" pod "17d55a46-9c7a-4cc7-989a-a27c45ee3e91" (UID: "17d55a46-9c7a-4cc7-989a-a27c45ee3e91"). InnerVolumeSpecName "kube-api-access-nf9q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:38:50.696626 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.694043 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-maas-api-tls" pod "17d55a46-9c7a-4cc7-989a-a27c45ee3e91" (UID: "17d55a46-9c7a-4cc7-989a-a27c45ee3e91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:38:50.703391 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.703331 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-metrics-certs" pod "17d55a46-9c7a-4cc7-989a-a27c45ee3e91" (UID: "17d55a46-9c7a-4cc7-989a-a27c45ee3e91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:38:50.772547 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.772505 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:50.772547 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.772544 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:50.772547 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.772554 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf9q4\" (UniqueName: \"kubernetes.io/projected/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-kube-api-access-nf9q4\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:50.772807 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:50.772564 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d55a46-9c7a-4cc7-989a-a27c45ee3e91-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:38:51.052606 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:51.052575 4409 generic.go:356] "Generic (PLEG): container finished" podID="17d55a46-9c7a-4cc7-989a-a27c45ee3e91" containerID="2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000" exitCode=0 Nov 03 07:38:51.052818 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:51.052623 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" Nov 03 07:38:51.052818 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:51.052668 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" event={"ID":"17d55a46-9c7a-4cc7-989a-a27c45ee3e91","Type":"ContainerDied","Data":"2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000"} Nov 03 07:38:51.052818 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:51.052741 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6" event={"ID":"17d55a46-9c7a-4cc7-989a-a27c45ee3e91","Type":"ContainerDied","Data":"46066e20ce4126bcba79a430b2e9aedac5d4525890c91d98dbc1a529bac119de"} Nov 03 07:38:51.052818 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:51.052767 4409 scope.go:122] "RemoveContainer" containerID="2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000" Nov 03 07:38:51.065105 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:51.065063 4409 scope.go:122] "RemoveContainer" containerID="2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000" Nov 03 07:38:51.065485 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:38:51.065456 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000\": container with ID starting with 2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000 not found: ID does not exist" containerID="2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000" Nov 03 07:38:51.065662 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:51.065491 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000"} err="failed to get container status \"2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000\": rpc error: code = NotFound desc = could not find container \"2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000\": container with ID starting with 2388a647c15adeed9a2806f011fb6ac083ff545901760771012523ac5c00c000 not found: ID does not exist" Nov 03 07:38:51.234748 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:51.234673 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6"] Nov 03 07:38:51.250778 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:51.250095 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-295fba7c-5cc5bd8c8c-thlt6"] Nov 03 07:38:52.915959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:52.915925 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d55a46-9c7a-4cc7-989a-a27c45ee3e91" path="/var/lib/kubelet/pods/17d55a46-9c7a-4cc7-989a-a27c45ee3e91/volumes" Nov 03 07:38:57.945213 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:38:57.945171 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz"] Nov 03 07:39:05.268963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:05.268926 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:39:05.276943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:05.276921 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:39:07.334582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:07.051827 4409 patch_prober.go:29] interesting pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:39:07.334582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:07.051890 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:39:07.334582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:07.271685 4409 generic.go:356] "Generic (PLEG): container finished" podID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerID="8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa" exitCode=0 Nov 03 07:39:07.334582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:07.271753 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" event={"ID":"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4","Type":"ContainerDied","Data":"8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa"} Nov 03 07:39:07.334582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:07.272144 4409 scope.go:122] "RemoveContainer" containerID="8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa" Nov 03 07:39:07.334582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:07.273645 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:39:07.334582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:07.273749 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:39:07.334582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:07.273803 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:39:08.025365 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.025281 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:39:08.297631 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.297595 4409 generic.go:356] "Generic (PLEG): container finished" podID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerID="9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e" exitCode=0 Nov 03 07:39:08.297733 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.297666 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" event={"ID":"d86da6dc-a180-4dd3-86e0-b34c28b47c37","Type":"ContainerDied","Data":"9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e"} Nov 03 07:39:08.298181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.298158 4409 scope.go:122] "RemoveContainer" containerID="9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e" Nov 03 07:39:08.299823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.299800 4409 generic.go:356] "Generic (PLEG): container finished" podID="25689a41-3fdd-401f-a99c-d59679618916" containerID="9cfec25f7888e1c10dbad56265d7919ef0c2179bdadcf2f4ee02a531777cdb65" exitCode=0 Nov 03 07:39:08.299904 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.299855 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" event={"ID":"25689a41-3fdd-401f-a99c-d59679618916","Type":"ContainerDied","Data":"9cfec25f7888e1c10dbad56265d7919ef0c2179bdadcf2f4ee02a531777cdb65"} Nov 03 07:39:08.299993 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.299964 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:39:08.300112 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.300048 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:39:08.300179 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.300119 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:39:08.300234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.300209 4409 scope.go:122] "RemoveContainer" containerID="9cfec25f7888e1c10dbad56265d7919ef0c2179bdadcf2f4ee02a531777cdb65" Nov 03 07:39:08.301854 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.301828 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:39:08.301937 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.301902 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:39:08.301978 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.301950 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:39:08.304196 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:08.304137 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" event={"ID":"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4","Type":"ContainerStarted","Data":"bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131"} Nov 03 07:39:08.840886 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:39:08.839014 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:39:09.063664 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:09.063633 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:39:09.064696 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:09.064663 4409 patch_prober.go:29] interesting pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 03 07:39:09.064886 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:09.064754 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 03 07:39:09.343742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:09.342743 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" event={"ID":"d86da6dc-a180-4dd3-86e0-b34c28b47c37","Type":"ContainerStarted","Data":"99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759"} Nov 03 07:39:09.347755 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:09.347698 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" event={"ID":"25689a41-3fdd-401f-a99c-d59679618916","Type":"ContainerStarted","Data":"fbb21adff0057e9b93c38f3341d192294bce0319a1ba099ea2cddfec910cda82"} Nov 03 07:39:09.738797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:09.735254 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:39:09.738797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:09.735389 4409 patch_prober.go:29] interesting pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 07:39:09.738797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:09.735431 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 07:39:09.939738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:09.925291 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:39:09.939738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:09.925401 4409 patch_prober.go:29] interesting pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 07:39:09.939738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:09.925444 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 07:39:10.063927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:10.063897 4409 patch_prober.go:29] interesting pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 03 07:39:10.064100 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:10.063942 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 03 07:39:10.752037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:10.751003 4409 patch_prober.go:29] interesting pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 07:39:10.752037 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:10.751079 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 07:39:10.949191 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:10.949146 4409 patch_prober.go:29] interesting pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 07:39:10.949706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:10.949217 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 07:39:11.064096 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:11.064066 4409 patch_prober.go:29] interesting pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 03 07:39:11.064278 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:11.064116 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 03 07:39:11.312739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:11.311755 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm"] Nov 03 07:39:11.332802 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:11.331770 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn"] Nov 03 07:39:11.351307 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:11.350406 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm"] Nov 03 07:39:11.740807 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:11.738027 4409 patch_prober.go:29] interesting pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 07:39:11.740807 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:11.738083 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 07:39:11.916134 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:11.916093 4409 patch_prober.go:29] interesting pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 07:39:11.916410 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:11.916382 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 07:39:12.064063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:12.064024 4409 patch_prober.go:29] interesting pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 03 07:39:12.064521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:12.064071 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 03 07:39:12.739634 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:12.739594 4409 patch_prober.go:29] interesting pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 07:39:12.740412 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:12.740365 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 07:39:12.910738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:12.909806 4409 patch_prober.go:29] interesting pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 07:39:12.910738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:12.909875 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 07:39:13.038795 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.035864 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" podUID="d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" containerName="main" containerID="cri-o://5f1a3ce551a74fd80c782abcd4df80e94f9bdaa624a39b49a7255339bf7a8c9d" gracePeriod=45 Nov 03 07:39:13.063848 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.063826 4409 patch_prober.go:29] interesting pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 03 07:39:13.063944 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.063878 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 03 07:39:13.212314 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.212246 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" podUID="d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": dial tcp 10.133.0.117:8000: connect: connection refused" Nov 03 07:39:13.641845 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.641479 4409 generic.go:356] "Generic (PLEG): container finished" podID="d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" containerID="5f1a3ce551a74fd80c782abcd4df80e94f9bdaa624a39b49a7255339bf7a8c9d" exitCode=0 Nov 03 07:39:13.641845 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.641698 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" event={"ID":"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db","Type":"ContainerDied","Data":"5f1a3ce551a74fd80c782abcd4df80e94f9bdaa624a39b49a7255339bf7a8c9d"} Nov 03 07:39:13.729917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.729812 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:39:13.735573 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.735539 4409 patch_prober.go:29] interesting pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 07:39:13.735678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.735588 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 07:39:13.777274 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.777242 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-home\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-home\") pod \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " Nov 03 07:39:13.777658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.777621 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-kube-api-access-dvkz4\" (UniqueName: \"kubernetes.io/projected/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-kube-api-access-dvkz4\") pod \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " Nov 03 07:39:13.778004 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.777986 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tls-certs\") pod \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " Nov 03 07:39:13.778173 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.778156 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-dshm\") pod \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " Nov 03 07:39:13.778328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.778310 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-model-cache\") pod \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " Nov 03 07:39:13.778603 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.778586 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tmp-dir\") pod \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\" (UID: \"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db\") " Nov 03 07:39:13.791070 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.791033 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-home" pod "d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" (UID: "d5ba6870-1fdb-4bb3-999e-c4e9b8f200db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:13.791582 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.791561 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tmp-dir" pod "d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" (UID: "d5ba6870-1fdb-4bb3-999e-c4e9b8f200db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:13.825109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.811596 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-kube-api-access-dvkz4" pod "d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" (UID: "d5ba6870-1fdb-4bb3-999e-c4e9b8f200db"). InnerVolumeSpecName "kube-api-access-dvkz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:39:13.825109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.812121 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tls-certs" pod "d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" (UID: "d5ba6870-1fdb-4bb3-999e-c4e9b8f200db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:39:13.825109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.813327 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-dshm" pod "d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" (UID: "d5ba6870-1fdb-4bb3-999e-c4e9b8f200db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:13.825109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.823954 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-model-cache" pod "d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" (UID: "d5ba6870-1fdb-4bb3-999e-c4e9b8f200db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:13.885771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.884192 4409 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tmp-dir\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:13.885771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.884226 4409 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-home\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:13.885771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.884242 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvkz4\" (UniqueName: \"kubernetes.io/projected/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-kube-api-access-dvkz4\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:13.885771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.884257 4409 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-tls-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:13.885771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.884271 4409 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-dshm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:13.885771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.884286 4409 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db-model-cache\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:13.928996 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.928904 4409 patch_prober.go:29] interesting pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 07:39:13.930844 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:13.930810 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 07:39:14.082759 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.080353 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:39:14.082759 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.081778 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:39:14.096003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.095980 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:39:14.650856 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.650805 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" event={"ID":"d5ba6870-1fdb-4bb3-999e-c4e9b8f200db","Type":"ContainerDied","Data":"a4e6ad60d1c7ad3ec20f2ea112b8254a94e7f7f729e3783080dfd8428df74dca"} Nov 03 07:39:14.651294 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.650883 4409 scope.go:122] "RemoveContainer" containerID="5f1a3ce551a74fd80c782abcd4df80e94f9bdaa624a39b49a7255339bf7a8c9d" Nov 03 07:39:14.651294 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.651105 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz" Nov 03 07:39:14.735652 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.735621 4409 patch_prober.go:29] interesting pod/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 03 07:39:14.735901 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.735877 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 03 07:39:14.798699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.798663 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:39:14.822968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.822930 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz"] Nov 03 07:39:14.828963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.828935 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-295fba7c/e2e-lifecycle-model-295fba7c-kserve-656b7ccf49-tshlz"] Nov 03 07:39:14.909479 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.909392 4409 patch_prober.go:29] interesting pod/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 03 07:39:14.909479 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.909450 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 03 07:39:14.917331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:14.917298 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" path="/var/lib/kubelet/pods/d5ba6870-1fdb-4bb3-999e-c4e9b8f200db/volumes" Nov 03 07:39:15.739561 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:15.739533 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:39:15.740028 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:15.739998 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:39:15.740645 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:15.740628 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:39:15.912269 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:15.912243 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:39:15.913360 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:15.913338 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:39:18.460813 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:18.460775 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp"] Nov 03 07:39:23.462000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:23.461962 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" podUID="f3c5b6de-bfce-4501-b6aa-44fb2359ae80" containerName="payload-processing" containerID="cri-o://625a296bfa749bb61b6ff360d918bb7690f89260803e3b49817b93697a7353de" gracePeriod=25 Nov 03 07:39:23.721618 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:23.721550 4409 generic.go:356] "Generic (PLEG): container finished" podID="f3c5b6de-bfce-4501-b6aa-44fb2359ae80" containerID="625a296bfa749bb61b6ff360d918bb7690f89260803e3b49817b93697a7353de" exitCode=0 Nov 03 07:39:23.721813 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:23.721623 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" event={"ID":"f3c5b6de-bfce-4501-b6aa-44fb2359ae80","Type":"ContainerDied","Data":"625a296bfa749bb61b6ff360d918bb7690f89260803e3b49817b93697a7353de"} Nov 03 07:39:23.721813 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:23.721675 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" event={"ID":"f3c5b6de-bfce-4501-b6aa-44fb2359ae80","Type":"ContainerDied","Data":"f95972177c14f268e939af9ede1e9cbc97ab19546e71a45a853926421785f008"} Nov 03 07:39:23.721813 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:23.721692 4409 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f95972177c14f268e939af9ede1e9cbc97ab19546e71a45a853926421785f008" Nov 03 07:39:23.727155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:23.727138 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" Nov 03 07:39:23.873792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:23.873761 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-kube-api-access-jx8fb\" (UniqueName: \"kubernetes.io/projected/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-kube-api-access-jx8fb\") pod \"f3c5b6de-bfce-4501-b6aa-44fb2359ae80\" (UID: \"f3c5b6de-bfce-4501-b6aa-44fb2359ae80\") " Nov 03 07:39:23.873977 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:23.873932 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-plugins-config-volume\") pod \"f3c5b6de-bfce-4501-b6aa-44fb2359ae80\" (UID: \"f3c5b6de-bfce-4501-b6aa-44fb2359ae80\") " Nov 03 07:39:23.874384 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:23.874351 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-plugins-config-volume" pod "f3c5b6de-bfce-4501-b6aa-44fb2359ae80" (UID: "f3c5b6de-bfce-4501-b6aa-44fb2359ae80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:39:23.874521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:23.874460 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:23.876689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:23.876662 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-kube-api-access-jx8fb" pod "f3c5b6de-bfce-4501-b6aa-44fb2359ae80" (UID: "f3c5b6de-bfce-4501-b6aa-44fb2359ae80"). InnerVolumeSpecName "kube-api-access-jx8fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:39:23.977405 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:23.976456 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx8fb\" (UniqueName: \"kubernetes.io/projected/f3c5b6de-bfce-4501-b6aa-44fb2359ae80-kube-api-access-jx8fb\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:24.728323 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:24.728292 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp" Nov 03 07:39:24.914963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:24.914929 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp"] Nov 03 07:39:24.915276 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:24.915256 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-3b41b9-597b56455c-k6xmp"] Nov 03 07:39:26.397751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.397686 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" containerID="cri-o://7e6c6e500d8aabaed579380e260b301e01854705b70e8a304889deba742b9baf" gracePeriod=45 Nov 03 07:39:26.459457 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.459426 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" containerID="cri-o://481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2" gracePeriod=45 Nov 03 07:39:26.463233 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.463208 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" containerID="cri-o://07cd6a070bf48bd270ebd77f58af7c2cb1706e63b89be5803163a8d8825ff86d" gracePeriod=45 Nov 03 07:39:26.590848 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.590809 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused" Nov 03 07:39:26.752901 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.752870 4409 generic.go:356] "Generic (PLEG): container finished" podID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerID="7e6c6e500d8aabaed579380e260b301e01854705b70e8a304889deba742b9baf" exitCode=0 Nov 03 07:39:26.753727 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.752949 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" event={"ID":"952827fe-e2c2-4471-bce0-07be8470d6c7","Type":"ContainerDied","Data":"7e6c6e500d8aabaed579380e260b301e01854705b70e8a304889deba742b9baf"} Nov 03 07:39:26.755554 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.755262 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:39:26.756871 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.756845 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" event={"ID":"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768","Type":"ContainerDied","Data":"07cd6a070bf48bd270ebd77f58af7c2cb1706e63b89be5803163a8d8825ff86d"} Nov 03 07:39:26.758033 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.756476 4409 generic.go:356] "Generic (PLEG): container finished" podID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerID="07cd6a070bf48bd270ebd77f58af7c2cb1706e63b89be5803163a8d8825ff86d" exitCode=0 Nov 03 07:39:26.759954 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.759933 4409 generic.go:356] "Generic (PLEG): container finished" podID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerID="481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2" exitCode=0 Nov 03 07:39:26.760059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.759964 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" event={"ID":"ee0451f1-c74f-491f-8ae6-9e6786e8ff34","Type":"ContainerDied","Data":"481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2"} Nov 03 07:39:26.760059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.759993 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" event={"ID":"ee0451f1-c74f-491f-8ae6-9e6786e8ff34","Type":"ContainerDied","Data":"e3704a7a7ea52cedaeb419dd05ef0f11dd33d2317755105d8884a4dde44656bc"} Nov 03 07:39:26.760059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.760001 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" Nov 03 07:39:26.760059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.760015 4409 scope.go:122] "RemoveContainer" containerID="481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2" Nov 03 07:39:26.770594 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.770569 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:39:26.772970 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.772945 4409 scope.go:122] "RemoveContainer" containerID="481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2" Nov 03 07:39:26.773233 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:39:26.773212 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2\": container with ID starting with 481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2 not found: ID does not exist" containerID="481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2" Nov 03 07:39:26.773323 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.773239 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2"} err="failed to get container status \"481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2\": rpc error: code = NotFound desc = could not find container \"481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2\": container with ID starting with 481a783fee5a7c7204b81ad5390dbc93d1c3467eaa501129a580a04392ea77c2 not found: ID does not exist" Nov 03 07:39:26.776521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.776503 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:39:26.913575 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.913547 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-dshm\") pod \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " Nov 03 07:39:26.913885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.913605 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-model-cache\") pod \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " Nov 03 07:39:26.913885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.913660 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-model-cache\") pod \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " Nov 03 07:39:26.913885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.913690 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tmp-dir\") pod \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " Nov 03 07:39:26.913885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.913756 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-home\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-home\") pod \"952827fe-e2c2-4471-bce0-07be8470d6c7\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " Nov 03 07:39:26.913885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.913825 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-model-cache\") pod \"952827fe-e2c2-4471-bce0-07be8470d6c7\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " Nov 03 07:39:26.913885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.913863 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-tmp-dir\") pod \"952827fe-e2c2-4471-bce0-07be8470d6c7\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " Nov 03 07:39:26.914398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.913936 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-dshm\") pod \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " Nov 03 07:39:26.914398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.913978 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-dshm\") pod \"952827fe-e2c2-4471-bce0-07be8470d6c7\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " Nov 03 07:39:26.914398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.914028 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/952827fe-e2c2-4471-bce0-07be8470d6c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/952827fe-e2c2-4471-bce0-07be8470d6c7-tls-certs\") pod \"952827fe-e2c2-4471-bce0-07be8470d6c7\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " Nov 03 07:39:26.914398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.914080 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-home\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-home\") pod \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " Nov 03 07:39:26.914398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.914120 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tmp-dir\") pod \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " Nov 03 07:39:26.914398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.914165 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tls-certs\") pod \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " Nov 03 07:39:26.914398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.914194 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-home\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-home\") pod \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " Nov 03 07:39:26.914398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.914248 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tls-certs\") pod \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " Nov 03 07:39:26.914398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.914315 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952827fe-e2c2-4471-bce0-07be8470d6c7-kube-api-access-lklsm\" (UniqueName: \"kubernetes.io/projected/952827fe-e2c2-4471-bce0-07be8470d6c7-kube-api-access-lklsm\") pod \"952827fe-e2c2-4471-bce0-07be8470d6c7\" (UID: \"952827fe-e2c2-4471-bce0-07be8470d6c7\") " Nov 03 07:39:26.914398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.914364 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-kube-api-access-qbrff\" (UniqueName: \"kubernetes.io/projected/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-kube-api-access-qbrff\") pod \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\" (UID: \"ee0451f1-c74f-491f-8ae6-9e6786e8ff34\") " Nov 03 07:39:26.915160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.914452 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-kube-api-access-s9w6l\" (UniqueName: \"kubernetes.io/projected/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-kube-api-access-s9w6l\") pod \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\" (UID: \"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768\") " Nov 03 07:39:26.916226 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.916194 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3c5b6de-bfce-4501-b6aa-44fb2359ae80" path="/var/lib/kubelet/pods/f3c5b6de-bfce-4501-b6aa-44fb2359ae80/volumes" Nov 03 07:39:26.916446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.916407 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-model-cache" pod "ee0451f1-c74f-491f-8ae6-9e6786e8ff34" (UID: "ee0451f1-c74f-491f-8ae6-9e6786e8ff34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.917148 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.917110 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-home" pod "7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" (UID: "7f2bd54c-5ad9-4521-8c53-71cbe7b2f768"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.918013 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.917873 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-home" pod "ee0451f1-c74f-491f-8ae6-9e6786e8ff34" (UID: "ee0451f1-c74f-491f-8ae6-9e6786e8ff34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.918125 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.918009 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tmp-dir" pod "7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" (UID: "7f2bd54c-5ad9-4521-8c53-71cbe7b2f768"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.918125 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.918016 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-tmp-dir" pod "952827fe-e2c2-4471-bce0-07be8470d6c7" (UID: "952827fe-e2c2-4471-bce0-07be8470d6c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.918605 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.918580 4409 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-model-cache\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:26.918698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.918607 4409 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tmp-dir\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:26.918698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.918621 4409 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-tmp-dir\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:26.918698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.918633 4409 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-home\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:26.918698 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.918645 4409 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-home\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:26.918940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.918872 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-model-cache" pod "952827fe-e2c2-4471-bce0-07be8470d6c7" (UID: "952827fe-e2c2-4471-bce0-07be8470d6c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.918940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.918922 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tmp-dir" pod "ee0451f1-c74f-491f-8ae6-9e6786e8ff34" (UID: "ee0451f1-c74f-491f-8ae6-9e6786e8ff34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.919042 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.918959 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-model-cache" pod "7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" (UID: "7f2bd54c-5ad9-4521-8c53-71cbe7b2f768"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.919042 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.918994 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-home" pod "952827fe-e2c2-4471-bce0-07be8470d6c7" (UID: "952827fe-e2c2-4471-bce0-07be8470d6c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.919793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.919771 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952827fe-e2c2-4471-bce0-07be8470d6c7-kube-api-access-lklsm" pod "952827fe-e2c2-4471-bce0-07be8470d6c7" (UID: "952827fe-e2c2-4471-bce0-07be8470d6c7"). InnerVolumeSpecName "kube-api-access-lklsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:39:26.920504 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.920409 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-dshm" pod "952827fe-e2c2-4471-bce0-07be8470d6c7" (UID: "952827fe-e2c2-4471-bce0-07be8470d6c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.928058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.928003 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/952827fe-e2c2-4471-bce0-07be8470d6c7-tls-certs" pod "952827fe-e2c2-4471-bce0-07be8470d6c7" (UID: "952827fe-e2c2-4471-bce0-07be8470d6c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:39:26.928058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.928019 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-dshm" pod "7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" (UID: "7f2bd54c-5ad9-4521-8c53-71cbe7b2f768"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:26.928058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.928034 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tls-certs" pod "ee0451f1-c74f-491f-8ae6-9e6786e8ff34" (UID: "ee0451f1-c74f-491f-8ae6-9e6786e8ff34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:39:26.928058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.928024 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-kube-api-access-s9w6l" pod "7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" (UID: "7f2bd54c-5ad9-4521-8c53-71cbe7b2f768"). InnerVolumeSpecName "kube-api-access-s9w6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:39:26.928058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.928039 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-kube-api-access-qbrff" pod "ee0451f1-c74f-491f-8ae6-9e6786e8ff34" (UID: "ee0451f1-c74f-491f-8ae6-9e6786e8ff34"). InnerVolumeSpecName "kube-api-access-qbrff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:39:26.928420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.928104 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tls-certs" pod "7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" (UID: "7f2bd54c-5ad9-4521-8c53-71cbe7b2f768"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:39:26.929511 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:26.929485 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-dshm" pod "ee0451f1-c74f-491f-8ae6-9e6786e8ff34" (UID: "ee0451f1-c74f-491f-8ae6-9e6786e8ff34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:39:27.019861 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019835 4409 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-dshm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.019861 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019859 4409 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-dshm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.019861 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019868 4409 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952827fe-e2c2-4471-bce0-07be8470d6c7-tls-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.019861 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019876 4409 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tmp-dir\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.020127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019884 4409 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-tls-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.020127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019893 4409 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-tls-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.020127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019903 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lklsm\" (UniqueName: \"kubernetes.io/projected/952827fe-e2c2-4471-bce0-07be8470d6c7-kube-api-access-lklsm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.020127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019913 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbrff\" (UniqueName: \"kubernetes.io/projected/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-kube-api-access-qbrff\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.020127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019921 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9w6l\" (UniqueName: \"kubernetes.io/projected/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-kube-api-access-s9w6l\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.020127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019930 4409 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0451f1-c74f-491f-8ae6-9e6786e8ff34-dshm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.020127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019938 4409 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768-model-cache\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.020127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019946 4409 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-home\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.020127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.019953 4409 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/952827fe-e2c2-4471-bce0-07be8470d6c7-model-cache\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:39:27.284701 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.284621 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm"] Nov 03 07:39:27.289969 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.289943 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm"] Nov 03 07:39:27.731598 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.731555 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-2-w6-3b41b9-kserve-66766cc67f-8b5pm" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 07:39:27.770786 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.770756 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" Nov 03 07:39:27.771121 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.770749 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm" event={"ID":"7f2bd54c-5ad9-4521-8c53-71cbe7b2f768","Type":"ContainerDied","Data":"2eeae48c81a91245f6dad17412ad6be7a13d04979154d7447ac89f77489e8654"} Nov 03 07:39:27.771256 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.771160 4409 scope.go:122] "RemoveContainer" containerID="07cd6a070bf48bd270ebd77f58af7c2cb1706e63b89be5803163a8d8825ff86d" Nov 03 07:39:27.793295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.793261 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" event={"ID":"952827fe-e2c2-4471-bce0-07be8470d6c7","Type":"ContainerDied","Data":"8c8c385381e51b75e474ac89912dff8bb6254b2ffd4cd451b5e218477559f1c5"} Nov 03 07:39:27.793446 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.793334 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn" Nov 03 07:39:27.803776 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.803757 4409 scope.go:122] "RemoveContainer" containerID="7e6c6e500d8aabaed579380e260b301e01854705b70e8a304889deba742b9baf" Nov 03 07:39:27.950941 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.950912 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn"] Nov 03 07:39:27.956476 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:27.956446 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/distinct-w6-3b41b9-kserve-84dd9b5dbd-n9nhn"] Nov 03 07:39:28.067980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:28.067944 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm"] Nov 03 07:39:28.073138 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:28.073116 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-3b41b9/premium-w6-3b41b9-kserve-66cb8b5689-j2pwm"] Nov 03 07:39:28.929867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:28.929830 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" path="/var/lib/kubelet/pods/7f2bd54c-5ad9-4521-8c53-71cbe7b2f768/volumes" Nov 03 07:39:28.937154 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:28.937123 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" path="/var/lib/kubelet/pods/952827fe-e2c2-4471-bce0-07be8470d6c7/volumes" Nov 03 07:39:28.938168 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:28.938142 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" path="/var/lib/kubelet/pods/ee0451f1-c74f-491f-8ae6-9e6786e8ff34/volumes" Nov 03 07:39:38.774090 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:39:38.774053 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:39:40.887380 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.887344 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s"] Nov 03 07:39:40.888089 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888071 4409 cpu_manager.go:414] "containerMap: removing container" podUID="d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" containerName="main" Nov 03 07:39:40.888089 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888090 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" containerName="main" Nov 03 07:39:40.888234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888139 4409 cpu_manager.go:414] "containerMap: removing container" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" Nov 03 07:39:40.888234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888144 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" Nov 03 07:39:40.888234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888153 4409 cpu_manager.go:414] "containerMap: removing container" podUID="f3c5b6de-bfce-4501-b6aa-44fb2359ae80" containerName="payload-processing" Nov 03 07:39:40.888234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888158 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c5b6de-bfce-4501-b6aa-44fb2359ae80" containerName="payload-processing" Nov 03 07:39:40.888234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888166 4409 cpu_manager.go:414] "containerMap: removing container" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" Nov 03 07:39:40.888234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888173 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" Nov 03 07:39:40.888234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888189 4409 cpu_manager.go:414] "containerMap: removing container" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" Nov 03 07:39:40.888234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888194 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" Nov 03 07:39:40.888234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888201 4409 cpu_manager.go:414] "containerMap: removing container" podUID="17d55a46-9c7a-4cc7-989a-a27c45ee3e91" containerName="maas-api" Nov 03 07:39:40.888234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888207 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d55a46-9c7a-4cc7-989a-a27c45ee3e91" containerName="maas-api" Nov 03 07:39:40.888234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888218 4409 cpu_manager.go:414] "containerMap: removing container" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" Nov 03 07:39:40.888234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888223 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" Nov 03 07:39:40.888608 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888322 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="a951fa97-ef86-4052-b96e-cf304f590ef6" containerName="istio-proxy" Nov 03 07:39:40.888608 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888333 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ba6870-1fdb-4bb3-999e-c4e9b8f200db" containerName="main" Nov 03 07:39:40.888608 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888346 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="952827fe-e2c2-4471-bce0-07be8470d6c7" containerName="main" Nov 03 07:39:40.888608 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888354 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c5b6de-bfce-4501-b6aa-44fb2359ae80" containerName="payload-processing" Nov 03 07:39:40.888608 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888360 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee0451f1-c74f-491f-8ae6-9e6786e8ff34" containerName="main" Nov 03 07:39:40.888608 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888366 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2bd54c-5ad9-4521-8c53-71cbe7b2f768" containerName="main" Nov 03 07:39:40.888608 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.888371 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d55a46-9c7a-4cc7-989a-a27c45ee3e91" containerName="maas-api" Nov 03 07:39:40.890921 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.890904 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" Nov 03 07:39:40.937897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.937859 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8sqz\" (UniqueName: \"kubernetes.io/projected/c2810d11-55ad-45fb-8a49-4c5760fd60e8-kube-api-access-t8sqz\") pod \"payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s\" (UID: \"c2810d11-55ad-45fb-8a49-4c5760fd60e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" Nov 03 07:39:40.938097 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.938064 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2810d11-55ad-45fb-8a49-4c5760fd60e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s\" (UID: \"c2810d11-55ad-45fb-8a49-4c5760fd60e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" Nov 03 07:39:40.950508 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.950479 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s"] Nov 03 07:39:41.014230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.998068 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-18b89b-dockercfg-km45j\"" Nov 03 07:39:41.014230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:40.998202 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-18b89b\"" Nov 03 07:39:41.042643 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.042598 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8sqz\" (UniqueName: \"kubernetes.io/projected/c2810d11-55ad-45fb-8a49-4c5760fd60e8-kube-api-access-t8sqz\") pod \"payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s\" (UID: \"c2810d11-55ad-45fb-8a49-4c5760fd60e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" Nov 03 07:39:41.069637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.059007 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2810d11-55ad-45fb-8a49-4c5760fd60e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s\" (UID: \"c2810d11-55ad-45fb-8a49-4c5760fd60e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" Nov 03 07:39:41.069637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.060150 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2810d11-55ad-45fb-8a49-4c5760fd60e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s\" (UID: \"c2810d11-55ad-45fb-8a49-4c5760fd60e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" Nov 03 07:39:41.143387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.143311 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8sqz\" (UniqueName: \"kubernetes.io/projected/c2810d11-55ad-45fb-8a49-4c5760fd60e8-kube-api-access-t8sqz\") pod \"payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s\" (UID: \"c2810d11-55ad-45fb-8a49-4c5760fd60e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" Nov 03 07:39:41.171051 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.170985 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm"] Nov 03 07:39:41.180900 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.176449 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm"] Nov 03 07:39:41.180900 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.176536 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" Nov 03 07:39:41.203837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.203792 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" Nov 03 07:39:41.261505 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.261465 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01d19dc-7c01-4944-bf95-792f185c575c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm\" (UID: \"b01d19dc-7c01-4944-bf95-792f185c575c\") " pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" Nov 03 07:39:41.261706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.261678 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfmhs\" (UniqueName: \"kubernetes.io/projected/b01d19dc-7c01-4944-bf95-792f185c575c-kube-api-access-nfmhs\") pod \"payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm\" (UID: \"b01d19dc-7c01-4944-bf95-792f185c575c\") " pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" Nov 03 07:39:41.362925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.362883 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01d19dc-7c01-4944-bf95-792f185c575c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm\" (UID: \"b01d19dc-7c01-4944-bf95-792f185c575c\") " pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" Nov 03 07:39:41.363105 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.363029 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfmhs\" (UniqueName: \"kubernetes.io/projected/b01d19dc-7c01-4944-bf95-792f185c575c-kube-api-access-nfmhs\") pod \"payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm\" (UID: \"b01d19dc-7c01-4944-bf95-792f185c575c\") " pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" Nov 03 07:39:41.363630 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.363603 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01d19dc-7c01-4944-bf95-792f185c575c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm\" (UID: \"b01d19dc-7c01-4944-bf95-792f185c575c\") " pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" Nov 03 07:39:41.426386 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.426306 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfmhs\" (UniqueName: \"kubernetes.io/projected/b01d19dc-7c01-4944-bf95-792f185c575c-kube-api-access-nfmhs\") pod \"payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm\" (UID: \"b01d19dc-7c01-4944-bf95-792f185c575c\") " pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" Nov 03 07:39:41.492514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.492484 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" Nov 03 07:39:41.596221 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.596189 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s"] Nov 03 07:39:41.597841 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:39:41.597802 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2810d11_55ad_45fb_8a49_4c5760fd60e8.slice/crio-cfed9e5b8804a3baa977b0de9cca3cd1760b68664a238e5af03393ef7b7b47a5 WatchSource:0}: Error finding container cfed9e5b8804a3baa977b0de9cca3cd1760b68664a238e5af03393ef7b7b47a5: Status 404 returned error can't find the container with id cfed9e5b8804a3baa977b0de9cca3cd1760b68664a238e5af03393ef7b7b47a5 Nov 03 07:39:41.872295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.872266 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm"] Nov 03 07:39:41.873227 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:39:41.873203 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01d19dc_7c01_4944_bf95_792f185c575c.slice/crio-744ed7ddb03a54415740b0b043c9376d6db334b96ce7038ad61f6ac482d48d79 WatchSource:0}: Error finding container 744ed7ddb03a54415740b0b043c9376d6db334b96ce7038ad61f6ac482d48d79: Status 404 returned error can't find the container with id 744ed7ddb03a54415740b0b043c9376d6db334b96ce7038ad61f6ac482d48d79 Nov 03 07:39:41.900649 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.900618 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" event={"ID":"c2810d11-55ad-45fb-8a49-4c5760fd60e8","Type":"ContainerStarted","Data":"368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b"} Nov 03 07:39:41.901059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.900677 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" event={"ID":"c2810d11-55ad-45fb-8a49-4c5760fd60e8","Type":"ContainerStarted","Data":"cfed9e5b8804a3baa977b0de9cca3cd1760b68664a238e5af03393ef7b7b47a5"} Nov 03 07:39:41.901059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.900700 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" Nov 03 07:39:41.901940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:41.901918 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" event={"ID":"b01d19dc-7c01-4944-bf95-792f185c575c","Type":"ContainerStarted","Data":"744ed7ddb03a54415740b0b043c9376d6db334b96ce7038ad61f6ac482d48d79"} Nov 03 07:39:42.020780 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:42.020676 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" podStartSLOduration=2.020656005 podStartE2EDuration="2.020656005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:39:42.018742254 +0000 UTC m=+2437.211283998" watchObservedRunningTime="2026-11-03 07:39:42.020656005 +0000 UTC m=+2437.213197944" Nov 03 07:39:42.922953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:42.922918 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" event={"ID":"b01d19dc-7c01-4944-bf95-792f185c575c","Type":"ContainerStarted","Data":"2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e"} Nov 03 07:39:43.040140 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:43.040079 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" podStartSLOduration=2.040062894 podStartE2EDuration="2.040062894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:39:43.03965259 +0000 UTC m=+2438.232194376" watchObservedRunningTime="2026-11-03 07:39:43.040062894 +0000 UTC m=+2438.232604644" Nov 03 07:39:43.925262 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:43.925232 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" Nov 03 07:39:44.207459 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.207364 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z"] Nov 03 07:39:44.210660 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.210637 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.267155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.267082 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-eac348-openshift-default-dockercfg-68mr7\"" Nov 03 07:39:44.305947 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.305851 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-token\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.306396 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.306370 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-socket\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.311796 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.311762 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkv4\" (UniqueName: \"kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-kube-api-access-lhkv4\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.312092 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.312059 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-credential-socket\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.312559 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.312536 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-certs\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.325397 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.325359 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-data\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.325735 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.325685 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istiod-ca-cert\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.325989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.325970 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-envoy\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.326200 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.326182 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-podinfo\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.387697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.387660 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z"] Nov 03 07:39:44.441460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.439882 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-certs\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.441460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.439948 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-data\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.441460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.439975 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istiod-ca-cert\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.441460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.440028 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-envoy\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.441460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.440053 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-podinfo\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.441460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.440149 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-token\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.441460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.440202 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-socket\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.441460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.440308 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkv4\" (UniqueName: \"kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-kube-api-access-lhkv4\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.441460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.440354 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-credential-socket\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.441460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.440920 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-credential-socket\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.441460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.441166 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-certs\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.441460 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.441373 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-data\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.442271 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.442002 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istiod-ca-cert\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.457792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.454179 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-socket\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.458016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.457948 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-envoy\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.467741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.467683 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-podinfo\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.514953 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.514920 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-token\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.521321 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.521294 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkv4\" (UniqueName: \"kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-kube-api-access-lhkv4\") pod \"e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.524196 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.524179 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:44.965766 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.965733 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z"] Nov 03 07:39:44.967171 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:39:44.967145 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1ee478_5863_4f95_8512_d6a2f5c4f6ed.slice/crio-2f543dc415afff22f342bdca3124562f5c95d5f38cf171ba8d27dbd2402ecf3a WatchSource:0}: Error finding container 2f543dc415afff22f342bdca3124562f5c95d5f38cf171ba8d27dbd2402ecf3a: Status 404 returned error can't find the container with id 2f543dc415afff22f342bdca3124562f5c95d5f38cf171ba8d27dbd2402ecf3a Nov 03 07:39:44.970064 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.970027 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:39:44.970161 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.970097 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:39:44.970161 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:44.970126 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:39:45.944563 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:45.944525 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" event={"ID":"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed","Type":"ContainerStarted","Data":"1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1"} Nov 03 07:39:45.944754 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:45.944588 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" event={"ID":"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed","Type":"ContainerStarted","Data":"2f543dc415afff22f342bdca3124562f5c95d5f38cf171ba8d27dbd2402ecf3a"} Nov 03 07:39:46.525275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:46.525245 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:46.531348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:46.531326 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:46.653927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:46.653874 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" podStartSLOduration=2.653861733 podStartE2EDuration="2.653861733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:39:46.133564751 +0000 UTC m=+2441.326106501" watchObservedRunningTime="2026-11-03 07:39:46.653861733 +0000 UTC m=+2441.846403482" Nov 03 07:39:46.954198 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:46.954166 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:46.955298 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:46.955272 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:39:52.921940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:52.921913 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" Nov 03 07:39:54.935295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:39:54.935267 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" Nov 03 07:40:05.194917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:05.194886 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw"] Nov 03 07:40:05.195342 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:05.195090 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" podUID="c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" containerName="istio-proxy" containerID="cri-o://5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434" gracePeriod=30 Nov 03 07:40:05.218266 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:05.218232 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd"] Nov 03 07:40:05.218475 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:05.218453 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" podUID="e0b5a99e-526e-4d5e-afc4-41beb52f6baa" containerName="istio-proxy" containerID="cri-o://8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce" gracePeriod=30 Nov 03 07:40:05.231702 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:05.231669 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r"] Nov 03 07:40:05.231963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:05.231937 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" podUID="08f8a61a-8d5d-43b8-b266-cdc49da02fb1" containerName="istio-proxy" containerID="cri-o://d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79" gracePeriod=30 Nov 03 07:40:07.428703 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:07.428672 4409 scope.go:122] "RemoveContainer" containerID="625a296bfa749bb61b6ff360d918bb7690f89260803e3b49817b93697a7353de" Nov 03 07:40:08.775574 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:40:08.775537 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:40:10.506969 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.506095 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:40:10.542512 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.542489 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:40:10.568574 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.568543 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:40:10.587065 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587042 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-envoy\") pod \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " Nov 03 07:40:10.587157 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587073 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-token\" (UniqueName: \"kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-token\") pod \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " Nov 03 07:40:10.587236 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587221 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-podinfo\") pod \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " Nov 03 07:40:10.587282 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587255 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-certs\") pod \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " Nov 03 07:40:10.587423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587407 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-socket\") pod \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " Nov 03 07:40:10.587486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587432 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-credential-socket\") pod \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " Nov 03 07:40:10.587486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587473 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-kube-api-access-ch7gc\" (UniqueName: \"kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-kube-api-access-ch7gc\") pod \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " Nov 03 07:40:10.587588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587524 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-certs" pod "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" (UID: "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.587588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587532 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istiod-ca-cert\") pod \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " Nov 03 07:40:10.587754 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587662 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-data\") pod \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\" (UID: \"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4\") " Nov 03 07:40:10.587938 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587822 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-socket" pod "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" (UID: "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.587996 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587966 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-credential-socket" pod "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" (UID: "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.587996 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.587983 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-data" pod "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" (UID: "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.588111 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.588060 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istiod-ca-cert" pod "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" (UID: "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:40:10.588449 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.588415 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.588449 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.588446 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.588649 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.588463 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.588649 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.588477 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.588649 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.588491 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.589515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.589479 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-envoy" pod "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" (UID: "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.589515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.589491 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-token" pod "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" (UID: "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.589764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.589747 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-podinfo" pod "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" (UID: "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:40:10.589834 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.589770 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-kube-api-access-ch7gc" pod "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" (UID: "c13f69f0-6a0a-4278-aedd-e2248f2f2cb4"). InnerVolumeSpecName "kube-api-access-ch7gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.689920 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.689895 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-kube-api-access-dwn8l\" (UniqueName: \"kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-kube-api-access-dwn8l\") pod \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " Nov 03 07:40:10.690081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.689952 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-envoy\") pod \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " Nov 03 07:40:10.690081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690000 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-certs\") pod \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " Nov 03 07:40:10.690081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690060 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-podinfo\") pod \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " Nov 03 07:40:10.690261 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690095 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-token\" (UniqueName: \"kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-token\") pod \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " Nov 03 07:40:10.690261 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690158 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-kube-api-access-qrncg\" (UniqueName: \"kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-kube-api-access-qrncg\") pod \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " Nov 03 07:40:10.690261 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690204 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istiod-ca-cert\") pod \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " Nov 03 07:40:10.690423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690259 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-envoy\") pod \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " Nov 03 07:40:10.690423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690310 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-credential-socket\") pod \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " Nov 03 07:40:10.690423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690347 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-certs" pod "08f8a61a-8d5d-43b8-b266-cdc49da02fb1" (UID: "08f8a61a-8d5d-43b8-b266-cdc49da02fb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.690423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690364 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-podinfo\") pod \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " Nov 03 07:40:10.690637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690475 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-socket\") pod \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " Nov 03 07:40:10.690637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690508 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-socket\") pod \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " Nov 03 07:40:10.690637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690544 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-token\" (UniqueName: \"kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-token\") pod \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " Nov 03 07:40:10.690637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690587 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-data\") pod \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " Nov 03 07:40:10.690637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690622 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-data\") pod \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " Nov 03 07:40:10.690958 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.690934 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-socket" pod "e0b5a99e-526e-4d5e-afc4-41beb52f6baa" (UID: "e0b5a99e-526e-4d5e-afc4-41beb52f6baa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.691148 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.691120 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-credential-socket" pod "e0b5a99e-526e-4d5e-afc4-41beb52f6baa" (UID: "e0b5a99e-526e-4d5e-afc4-41beb52f6baa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.691397 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.691363 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istiod-ca-cert" pod "e0b5a99e-526e-4d5e-afc4-41beb52f6baa" (UID: "e0b5a99e-526e-4d5e-afc4-41beb52f6baa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:40:10.692085 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.692056 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-data" pod "08f8a61a-8d5d-43b8-b266-cdc49da02fb1" (UID: "08f8a61a-8d5d-43b8-b266-cdc49da02fb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.692295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.692268 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-socket" pod "08f8a61a-8d5d-43b8-b266-cdc49da02fb1" (UID: "08f8a61a-8d5d-43b8-b266-cdc49da02fb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.692411 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.692387 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-data" pod "e0b5a99e-526e-4d5e-afc4-41beb52f6baa" (UID: "e0b5a99e-526e-4d5e-afc4-41beb52f6baa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.692565 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.692424 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-certs\") pod \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\" (UID: \"e0b5a99e-526e-4d5e-afc4-41beb52f6baa\") " Nov 03 07:40:10.692565 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.692465 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istiod-ca-cert\") pod \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " Nov 03 07:40:10.692565 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.692505 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-credential-socket\") pod \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\" (UID: \"08f8a61a-8d5d-43b8-b266-cdc49da02fb1\") " Nov 03 07:40:10.692754 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.692651 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-certs" pod "e0b5a99e-526e-4d5e-afc4-41beb52f6baa" (UID: "e0b5a99e-526e-4d5e-afc4-41beb52f6baa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.692865 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.692840 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-credential-socket" pod "08f8a61a-8d5d-43b8-b266-cdc49da02fb1" (UID: "08f8a61a-8d5d-43b8-b266-cdc49da02fb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.692951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.692879 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istiod-ca-cert" pod "08f8a61a-8d5d-43b8-b266-cdc49da02fb1" (UID: "08f8a61a-8d5d-43b8-b266-cdc49da02fb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:40:10.693216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693193 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693222 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7gc\" (UniqueName: \"kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-kube-api-access-ch7gc\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693238 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693251 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693265 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693278 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693291 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693306 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693318 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693332 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693346 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693361 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693376 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693389 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.693693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.693426 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-envoy" pod "08f8a61a-8d5d-43b8-b266-cdc49da02fb1" (UID: "08f8a61a-8d5d-43b8-b266-cdc49da02fb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.694963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.694941 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-token" pod "08f8a61a-8d5d-43b8-b266-cdc49da02fb1" (UID: "08f8a61a-8d5d-43b8-b266-cdc49da02fb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.695766 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.695728 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-podinfo" pod "e0b5a99e-526e-4d5e-afc4-41beb52f6baa" (UID: "e0b5a99e-526e-4d5e-afc4-41beb52f6baa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:40:10.695863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.695777 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-podinfo" pod "08f8a61a-8d5d-43b8-b266-cdc49da02fb1" (UID: "08f8a61a-8d5d-43b8-b266-cdc49da02fb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:40:10.695863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.695782 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-envoy" pod "e0b5a99e-526e-4d5e-afc4-41beb52f6baa" (UID: "e0b5a99e-526e-4d5e-afc4-41beb52f6baa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:10.695863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.695779 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-kube-api-access-qrncg" pod "e0b5a99e-526e-4d5e-afc4-41beb52f6baa" (UID: "e0b5a99e-526e-4d5e-afc4-41beb52f6baa"). InnerVolumeSpecName "kube-api-access-qrncg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.695863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.695813 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-kube-api-access-dwn8l" pod "08f8a61a-8d5d-43b8-b266-cdc49da02fb1" (UID: "08f8a61a-8d5d-43b8-b266-cdc49da02fb1"). InnerVolumeSpecName "kube-api-access-dwn8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.697349 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.697330 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-token" pod "e0b5a99e-526e-4d5e-afc4-41beb52f6baa" (UID: "e0b5a99e-526e-4d5e-afc4-41beb52f6baa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:10.795521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.795439 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwn8l\" (UniqueName: \"kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-kube-api-access-dwn8l\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.795521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.795484 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.795521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.795501 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.795521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.795515 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.795790 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.795529 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrncg\" (UniqueName: \"kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-kube-api-access-qrncg\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.795790 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.795543 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.795790 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.795557 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f8a61a-8d5d-43b8-b266-cdc49da02fb1-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:10.795790 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:10.795570 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b5a99e-526e-4d5e-afc4-41beb52f6baa-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:11.222061 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.222026 4409 generic.go:356] "Generic (PLEG): container finished" podID="c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" containerID="5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434" exitCode=0 Nov 03 07:40:11.222249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.222088 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" Nov 03 07:40:11.222249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.222117 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" event={"ID":"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4","Type":"ContainerDied","Data":"5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434"} Nov 03 07:40:11.222249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.222202 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw" event={"ID":"c13f69f0-6a0a-4278-aedd-e2248f2f2cb4","Type":"ContainerDied","Data":"3aabb0fac1bb59b42e6db89be46d4f77aa3300f4906bc5aba35aba64fa210c0a"} Nov 03 07:40:11.222249 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.222236 4409 scope.go:122] "RemoveContainer" containerID="5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434" Nov 03 07:40:11.223845 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.223825 4409 generic.go:356] "Generic (PLEG): container finished" podID="08f8a61a-8d5d-43b8-b266-cdc49da02fb1" containerID="d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79" exitCode=0 Nov 03 07:40:11.223940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.223862 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" Nov 03 07:40:11.223940 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.223909 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" event={"ID":"08f8a61a-8d5d-43b8-b266-cdc49da02fb1","Type":"ContainerDied","Data":"d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79"} Nov 03 07:40:11.224054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.223951 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r" event={"ID":"08f8a61a-8d5d-43b8-b266-cdc49da02fb1","Type":"ContainerDied","Data":"9696dd9711a35bf781ffa74575ccdf4e31160cc63b76e566eef59f412481a74a"} Nov 03 07:40:11.226222 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.226091 4409 generic.go:356] "Generic (PLEG): container finished" podID="e0b5a99e-526e-4d5e-afc4-41beb52f6baa" containerID="8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce" exitCode=0 Nov 03 07:40:11.226318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.226261 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" event={"ID":"e0b5a99e-526e-4d5e-afc4-41beb52f6baa","Type":"ContainerDied","Data":"8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce"} Nov 03 07:40:11.226379 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.226312 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" event={"ID":"e0b5a99e-526e-4d5e-afc4-41beb52f6baa","Type":"ContainerDied","Data":"53bed7a8a015fa250df689cf399d39252cac015ad5823193196e00d057ff3b04"} Nov 03 07:40:11.226431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.226386 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd" Nov 03 07:40:11.244912 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.244874 4409 scope.go:122] "RemoveContainer" containerID="5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434" Nov 03 07:40:11.245190 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:40:11.245171 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434\": container with ID starting with 5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434 not found: ID does not exist" containerID="5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434" Nov 03 07:40:11.245253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.245202 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434"} err="failed to get container status \"5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434\": rpc error: code = NotFound desc = could not find container \"5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434\": container with ID starting with 5017b48b6a8ec7732e2c3ae27c510604984a6a4f9efa6eb71417854ce18e2434 not found: ID does not exist" Nov 03 07:40:11.245253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.245224 4409 scope.go:122] "RemoveContainer" containerID="d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79" Nov 03 07:40:11.254281 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.254258 4409 scope.go:122] "RemoveContainer" containerID="d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79" Nov 03 07:40:11.254550 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:40:11.254530 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79\": container with ID starting with d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79 not found: ID does not exist" containerID="d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79" Nov 03 07:40:11.254613 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.254556 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79"} err="failed to get container status \"d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79\": rpc error: code = NotFound desc = could not find container \"d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79\": container with ID starting with d282ba3ecb89077e8c136c646b7be2c3b5ac58777eade16dbbf322dbf85c7f79 not found: ID does not exist" Nov 03 07:40:11.254613 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.254572 4409 scope.go:122] "RemoveContainer" containerID="8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce" Nov 03 07:40:11.263739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.263699 4409 scope.go:122] "RemoveContainer" containerID="8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce" Nov 03 07:40:11.263988 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:40:11.263967 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce\": container with ID starting with 8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce not found: ID does not exist" containerID="8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce" Nov 03 07:40:11.264057 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.263994 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce"} err="failed to get container status \"8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce\": rpc error: code = NotFound desc = could not find container \"8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce\": container with ID starting with 8ffa0291cfa304f903ed95e5eddf597add9835496213b84d0c6f1d59e373ccce not found: ID does not exist" Nov 03 07:40:11.392824 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.392795 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r"] Nov 03 07:40:11.441576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.441549 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-mkg2r"] Nov 03 07:40:11.510094 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.510033 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw"] Nov 03 07:40:11.516248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.516225 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-jtxtw"] Nov 03 07:40:11.628258 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.628236 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd"] Nov 03 07:40:11.633982 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:11.633961 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-295fba7c-openshift-default-6b878475cc-bt6vd"] Nov 03 07:40:12.161789 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.161756 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58"] Nov 03 07:40:12.161985 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.161963 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" podUID="41668674-15da-4311-a5fb-4254db00797e" containerName="maas-api" containerID="cri-o://3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19" gracePeriod=30 Nov 03 07:40:12.484674 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.484649 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:40:12.629529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.619426 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-maas-api-tls\") pod \"41668674-15da-4311-a5fb-4254db00797e\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " Nov 03 07:40:12.629529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.619485 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41668674-15da-4311-a5fb-4254db00797e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41668674-15da-4311-a5fb-4254db00797e-openshift-service-ca\") pod \"41668674-15da-4311-a5fb-4254db00797e\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " Nov 03 07:40:12.629529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.619597 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-metrics-certs\") pod \"41668674-15da-4311-a5fb-4254db00797e\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " Nov 03 07:40:12.629529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.619777 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41668674-15da-4311-a5fb-4254db00797e-kube-api-access-rhhmn\" (UniqueName: \"kubernetes.io/projected/41668674-15da-4311-a5fb-4254db00797e-kube-api-access-rhhmn\") pod \"41668674-15da-4311-a5fb-4254db00797e\" (UID: \"41668674-15da-4311-a5fb-4254db00797e\") " Nov 03 07:40:12.629529 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.622503 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41668674-15da-4311-a5fb-4254db00797e-openshift-service-ca" pod "41668674-15da-4311-a5fb-4254db00797e" (UID: "41668674-15da-4311-a5fb-4254db00797e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:40:12.633945 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.633914 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-maas-api-tls" pod "41668674-15da-4311-a5fb-4254db00797e" (UID: "41668674-15da-4311-a5fb-4254db00797e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:40:12.634985 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.634958 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41668674-15da-4311-a5fb-4254db00797e-kube-api-access-rhhmn" pod "41668674-15da-4311-a5fb-4254db00797e" (UID: "41668674-15da-4311-a5fb-4254db00797e"). InnerVolumeSpecName "kube-api-access-rhhmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:12.635369 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.635346 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-metrics-certs" pod "41668674-15da-4311-a5fb-4254db00797e" (UID: "41668674-15da-4311-a5fb-4254db00797e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:40:12.729903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.728651 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:12.729903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.728691 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41668674-15da-4311-a5fb-4254db00797e-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:12.729903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.728705 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41668674-15da-4311-a5fb-4254db00797e-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:12.729903 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.728741 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhhmn\" (UniqueName: \"kubernetes.io/projected/41668674-15da-4311-a5fb-4254db00797e-kube-api-access-rhhmn\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:12.913869 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.913840 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f8a61a-8d5d-43b8-b266-cdc49da02fb1" path="/var/lib/kubelet/pods/08f8a61a-8d5d-43b8-b266-cdc49da02fb1/volumes" Nov 03 07:40:12.914347 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.914334 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" path="/var/lib/kubelet/pods/c13f69f0-6a0a-4278-aedd-e2248f2f2cb4/volumes" Nov 03 07:40:12.914758 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:12.914746 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b5a99e-526e-4d5e-afc4-41beb52f6baa" path="/var/lib/kubelet/pods/e0b5a99e-526e-4d5e-afc4-41beb52f6baa/volumes" Nov 03 07:40:13.246225 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:13.246190 4409 generic.go:356] "Generic (PLEG): container finished" podID="41668674-15da-4311-a5fb-4254db00797e" containerID="3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19" exitCode=0 Nov 03 07:40:13.246431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:13.246246 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" Nov 03 07:40:13.246431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:13.246256 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" event={"ID":"41668674-15da-4311-a5fb-4254db00797e","Type":"ContainerDied","Data":"3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19"} Nov 03 07:40:13.246431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:13.246312 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58" event={"ID":"41668674-15da-4311-a5fb-4254db00797e","Type":"ContainerDied","Data":"9d15aa5e601b6791b2dcb371512273b71a4d8f0a5152b188d649e52a5bffbb3f"} Nov 03 07:40:13.246431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:13.246339 4409 scope.go:122] "RemoveContainer" containerID="3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19" Nov 03 07:40:13.256453 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:13.256435 4409 scope.go:122] "RemoveContainer" containerID="3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19" Nov 03 07:40:13.256851 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:40:13.256827 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19\": container with ID starting with 3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19 not found: ID does not exist" containerID="3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19" Nov 03 07:40:13.256952 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:13.256857 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19"} err="failed to get container status \"3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19\": rpc error: code = NotFound desc = could not find container \"3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19\": container with ID starting with 3ae5ad1f85c8d1bfe12136b35323328b333d280bb6340357d46f69d1abf8cd19 not found: ID does not exist" Nov 03 07:40:13.414882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:13.414851 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58"] Nov 03 07:40:13.464545 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:13.464521 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3fea3a-8c5fd889f-fxn58"] Nov 03 07:40:14.913851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:14.913813 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41668674-15da-4311-a5fb-4254db00797e" path="/var/lib/kubelet/pods/41668674-15da-4311-a5fb-4254db00797e/volumes" Nov 03 07:40:22.562073 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:22.561994 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s"] Nov 03 07:40:22.562528 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:22.562281 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" containerID="cri-o://33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67" gracePeriod=30 Nov 03 07:40:22.637418 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:22.637384 4409 patch_prober.go:29] interesting pod/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:40:22.637801 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:22.637762 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:40:27.849732 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.849623 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:40:27.931471 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.931429 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-credential-socket\") pod \"fbee391f-f81b-4975-aa4d-112fd30e5422\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " Nov 03 07:40:27.931678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.931506 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-envoy\") pod \"fbee391f-f81b-4975-aa4d-112fd30e5422\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " Nov 03 07:40:27.931678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.931563 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-data\") pod \"fbee391f-f81b-4975-aa4d-112fd30e5422\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " Nov 03 07:40:27.931678 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.931635 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-socket\") pod \"fbee391f-f81b-4975-aa4d-112fd30e5422\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " Nov 03 07:40:27.931926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.931767 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-token\" (UniqueName: \"kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-token\") pod \"fbee391f-f81b-4975-aa4d-112fd30e5422\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " Nov 03 07:40:27.931926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.931788 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-credential-socket" pod "fbee391f-f81b-4975-aa4d-112fd30e5422" (UID: "fbee391f-f81b-4975-aa4d-112fd30e5422"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:27.931926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.931801 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-kube-api-access-nt85x\" (UniqueName: \"kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-kube-api-access-nt85x\") pod \"fbee391f-f81b-4975-aa4d-112fd30e5422\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " Nov 03 07:40:27.931926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.931855 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-data" pod "fbee391f-f81b-4975-aa4d-112fd30e5422" (UID: "fbee391f-f81b-4975-aa4d-112fd30e5422"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:27.931926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.931883 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-certs\") pod \"fbee391f-f81b-4975-aa4d-112fd30e5422\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " Nov 03 07:40:27.932215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.931949 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-podinfo\") pod \"fbee391f-f81b-4975-aa4d-112fd30e5422\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " Nov 03 07:40:27.932215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.932003 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbee391f-f81b-4975-aa4d-112fd30e5422-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbee391f-f81b-4975-aa4d-112fd30e5422-istiod-ca-cert\") pod \"fbee391f-f81b-4975-aa4d-112fd30e5422\" (UID: \"fbee391f-f81b-4975-aa4d-112fd30e5422\") " Nov 03 07:40:27.932215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.932198 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-certs" pod "fbee391f-f81b-4975-aa4d-112fd30e5422" (UID: "fbee391f-f81b-4975-aa4d-112fd30e5422"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:27.932623 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.932598 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:27.932762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.932629 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:27.932762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.932644 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:27.932762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.932643 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbee391f-f81b-4975-aa4d-112fd30e5422-istiod-ca-cert" pod "fbee391f-f81b-4975-aa4d-112fd30e5422" (UID: "fbee391f-f81b-4975-aa4d-112fd30e5422"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:40:27.932762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.932662 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-socket" pod "fbee391f-f81b-4975-aa4d-112fd30e5422" (UID: "fbee391f-f81b-4975-aa4d-112fd30e5422"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:27.934207 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.934173 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-envoy" pod "fbee391f-f81b-4975-aa4d-112fd30e5422" (UID: "fbee391f-f81b-4975-aa4d-112fd30e5422"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:40:27.934329 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.934256 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-kube-api-access-nt85x" pod "fbee391f-f81b-4975-aa4d-112fd30e5422" (UID: "fbee391f-f81b-4975-aa4d-112fd30e5422"). InnerVolumeSpecName "kube-api-access-nt85x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:27.934627 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.934604 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-token" pod "fbee391f-f81b-4975-aa4d-112fd30e5422" (UID: "fbee391f-f81b-4975-aa4d-112fd30e5422"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:40:27.935086 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:27.935065 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-podinfo" pod "fbee391f-f81b-4975-aa4d-112fd30e5422" (UID: "fbee391f-f81b-4975-aa4d-112fd30e5422"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:40:28.037673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.037632 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:28.037673 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.037673 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbee391f-f81b-4975-aa4d-112fd30e5422-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:28.037989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.037687 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:28.037989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.037700 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbee391f-f81b-4975-aa4d-112fd30e5422-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:28.037989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.037737 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:28.037989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.037753 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt85x\" (UniqueName: \"kubernetes.io/projected/fbee391f-f81b-4975-aa4d-112fd30e5422-kube-api-access-nt85x\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:40:28.347597 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.347553 4409 generic.go:356] "Generic (PLEG): container finished" podID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerID="33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67" exitCode=0 Nov 03 07:40:28.347831 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.347668 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" Nov 03 07:40:28.347928 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.347899 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" event={"ID":"fbee391f-f81b-4975-aa4d-112fd30e5422","Type":"ContainerDied","Data":"33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67"} Nov 03 07:40:28.347997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.347961 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s" event={"ID":"fbee391f-f81b-4975-aa4d-112fd30e5422","Type":"ContainerDied","Data":"64e7f713da4160d187016d370b0d354f381e27f260173ac07aad1b04db108db2"} Nov 03 07:40:28.348059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.347997 4409 scope.go:122] "RemoveContainer" containerID="33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67" Nov 03 07:40:28.365896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.365866 4409 scope.go:122] "RemoveContainer" containerID="fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43" Nov 03 07:40:28.384251 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.383137 4409 scope.go:122] "RemoveContainer" containerID="33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67" Nov 03 07:40:28.384251 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:40:28.383570 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67\": container with ID starting with 33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67 not found: ID does not exist" containerID="33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67" Nov 03 07:40:28.384251 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.383605 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67"} err="failed to get container status \"33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67\": rpc error: code = NotFound desc = could not find container \"33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67\": container with ID starting with 33072baaf92416010353b6a79ffa6f6ac0e53bc8119799a42c394e027a687b67 not found: ID does not exist" Nov 03 07:40:28.384251 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.383632 4409 scope.go:122] "RemoveContainer" containerID="fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43" Nov 03 07:40:28.384251 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:40:28.384146 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43\": container with ID starting with fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43 not found: ID does not exist" containerID="fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43" Nov 03 07:40:28.384251 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.384175 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43"} err="failed to get container status \"fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43\": rpc error: code = NotFound desc = could not find container \"fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43\": container with ID starting with fe45b51770af1506465bd0db25702cc35d138e463c5a306e891ae1c160a97e43 not found: ID does not exist" Nov 03 07:40:28.524106 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.524065 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s"] Nov 03 07:40:28.573706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.573665 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3fea3a-openshift-default-7df4bff97-xm75s"] Nov 03 07:40:28.919699 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:28.919664 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" path="/var/lib/kubelet/pods/fbee391f-f81b-4975-aa4d-112fd30e5422/volumes" Nov 03 07:40:34.833221 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.833179 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j"] Nov 03 07:40:34.834091 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834068 4409 cpu_manager.go:414] "containerMap: removing container" podUID="41668674-15da-4311-a5fb-4254db00797e" containerName="maas-api" Nov 03 07:40:34.834155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834095 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41668674-15da-4311-a5fb-4254db00797e" containerName="maas-api" Nov 03 07:40:34.834155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834114 4409 cpu_manager.go:414] "containerMap: removing container" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" Nov 03 07:40:34.834155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834122 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" Nov 03 07:40:34.834155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834141 4409 cpu_manager.go:414] "containerMap: removing container" podUID="c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" containerName="istio-proxy" Nov 03 07:40:34.834155 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834149 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" containerName="istio-proxy" Nov 03 07:40:34.834317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834161 4409 cpu_manager.go:414] "containerMap: removing container" podUID="e0b5a99e-526e-4d5e-afc4-41beb52f6baa" containerName="istio-proxy" Nov 03 07:40:34.834317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834167 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b5a99e-526e-4d5e-afc4-41beb52f6baa" containerName="istio-proxy" Nov 03 07:40:34.834317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834224 4409 cpu_manager.go:414] "containerMap: removing container" podUID="08f8a61a-8d5d-43b8-b266-cdc49da02fb1" containerName="istio-proxy" Nov 03 07:40:34.834317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834231 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f8a61a-8d5d-43b8-b266-cdc49da02fb1" containerName="istio-proxy" Nov 03 07:40:34.834433 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834392 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" Nov 03 07:40:34.834433 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834406 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f8a61a-8d5d-43b8-b266-cdc49da02fb1" containerName="istio-proxy" Nov 03 07:40:34.834433 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834420 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="41668674-15da-4311-a5fb-4254db00797e" containerName="maas-api" Nov 03 07:40:34.834433 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834430 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" Nov 03 07:40:34.834554 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834441 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13f69f0-6a0a-4278-aedd-e2248f2f2cb4" containerName="istio-proxy" Nov 03 07:40:34.834554 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.834450 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b5a99e-526e-4d5e-afc4-41beb52f6baa" containerName="istio-proxy" Nov 03 07:40:34.837391 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.837372 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:34.942906 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.942876 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-18b89b-serving-cert\"" Nov 03 07:40:34.946054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.946030 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j"] Nov 03 07:40:34.978656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.978623 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1401ab1-a79a-43dc-9321-60679a32ca36-openshift-service-ca\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:34.978853 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.978678 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-metrics-certs\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:34.978936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.978877 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-maas-api-tls\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:34.979003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:34.978947 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd7gg\" (UniqueName: \"kubernetes.io/projected/b1401ab1-a79a-43dc-9321-60679a32ca36-kube-api-access-qd7gg\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:35.084621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:35.084526 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1401ab1-a79a-43dc-9321-60679a32ca36-openshift-service-ca\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:35.085885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:35.085863 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-metrics-certs\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:35.087202 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:35.086875 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-maas-api-tls\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:35.095044 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:35.095014 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd7gg\" (UniqueName: \"kubernetes.io/projected/b1401ab1-a79a-43dc-9321-60679a32ca36-kube-api-access-qd7gg\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:35.095781 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:40:35.090367 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-serving-cert: secret "maas-api-e2e-api-b-18b89b-serving-cert" not found Nov 03 07:40:35.096378 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:40:35.096358 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-maas-api-tls podName:b1401ab1-a79a-43dc-9321-60679a32ca36 nodeName:}" failed. No retries permitted until 2026-11-03 07:40:35.596332584 +0000 UTC m=+2490.788874330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-maas-api-tls") pod "maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" (UID: "b1401ab1-a79a-43dc-9321-60679a32ca36") : secret "maas-api-e2e-api-b-18b89b-serving-cert" not found Nov 03 07:40:35.096746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:35.085749 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1401ab1-a79a-43dc-9321-60679a32ca36-openshift-service-ca\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:35.108407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:35.108375 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-metrics-certs\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:35.161573 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:35.161539 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd7gg\" (UniqueName: \"kubernetes.io/projected/b1401ab1-a79a-43dc-9321-60679a32ca36-kube-api-access-qd7gg\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:35.603875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:35.603830 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-maas-api-tls\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:35.606472 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:35.606446 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-maas-api-tls\") pod \"maas-api-e2e-api-b-18b89b-595795b66b-9fq9j\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:35.752209 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:35.752181 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:36.135237 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:36.135207 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j"] Nov 03 07:40:36.136330 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:40:36.136302 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1401ab1_a79a_43dc_9321_60679a32ca36.slice/crio-cd83b70b09577817ed29a432a95dbfec9dd341937d111bb881f4569de8da9afe WatchSource:0}: Error finding container cd83b70b09577817ed29a432a95dbfec9dd341937d111bb881f4569de8da9afe: Status 404 returned error can't find the container with id cd83b70b09577817ed29a432a95dbfec9dd341937d111bb881f4569de8da9afe Nov 03 07:40:36.137782 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:36.137763 4409 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:40:36.401512 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:36.401422 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" event={"ID":"b1401ab1-a79a-43dc-9321-60679a32ca36","Type":"ContainerStarted","Data":"cd83b70b09577817ed29a432a95dbfec9dd341937d111bb881f4569de8da9afe"} Nov 03 07:40:37.410809 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:37.410758 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" event={"ID":"b1401ab1-a79a-43dc-9321-60679a32ca36","Type":"ContainerStarted","Data":"d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab"} Nov 03 07:40:37.411253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:37.410968 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:37.531163 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:37.531106 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" podStartSLOduration=2.682357047 podStartE2EDuration="3.531091455s" totalImagesPullingTime="848.734408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:40:37.529381144 +0000 UTC m=+2492.721922917" watchObservedRunningTime="2026-11-03 07:40:37.531091455 +0000 UTC m=+2492.723633239" Nov 03 07:40:38.776076 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:40:38.776039 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:40:43.423514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:43.423483 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:40:44.913511 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:44.913480 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b"] Nov 03 07:40:44.914087 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:44.914073 4409 cpu_manager.go:414] "containerMap: removing container" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" Nov 03 07:40:44.914134 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:44.914089 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbee391f-f81b-4975-aa4d-112fd30e5422" containerName="istio-proxy" Nov 03 07:40:44.917494 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:44.917479 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.021651 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.021617 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-44c08f-serving-cert\"" Nov 03 07:40:45.024857 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.024832 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b"] Nov 03 07:40:45.070428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.070396 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-openshift-service-ca\") pod \"maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.070596 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.070442 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-maas-api-tls\") pod \"maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.070596 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.070538 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg78v\" (UniqueName: \"kubernetes.io/projected/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-kube-api-access-lg78v\") pod \"maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.070776 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.070751 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-metrics-certs\") pod \"maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.172352 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.172262 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-metrics-certs\") pod \"maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.172513 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.172439 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-openshift-service-ca\") pod \"maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.172513 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.172492 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-maas-api-tls\") pod \"maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.172621 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.172533 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg78v\" (UniqueName: \"kubernetes.io/projected/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-kube-api-access-lg78v\") pod \"maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.173269 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.173152 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-openshift-service-ca\") pod \"maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.175063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.175042 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-metrics-certs\") pod \"maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.175186 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.175169 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-maas-api-tls\") pod \"maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.236747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.236700 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg78v\" (UniqueName: \"kubernetes.io/projected/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-kube-api-access-lg78v\") pod \"maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.531912 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.531824 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:45.913992 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:45.913963 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b"] Nov 03 07:40:45.915458 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:40:45.915419 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0432b16_3bc9_42dc_84b6_b685dd7d0a88.slice/crio-de48b6c02641bd79714e8400df31ec4ed4715e93040afe6f092af761cd1aa340 WatchSource:0}: Error finding container de48b6c02641bd79714e8400df31ec4ed4715e93040afe6f092af761cd1aa340: Status 404 returned error can't find the container with id de48b6c02641bd79714e8400df31ec4ed4715e93040afe6f092af761cd1aa340 Nov 03 07:40:46.472866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:46.472824 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" event={"ID":"b0432b16-3bc9-42dc-84b6-b685dd7d0a88","Type":"ContainerStarted","Data":"de48b6c02641bd79714e8400df31ec4ed4715e93040afe6f092af761cd1aa340"} Nov 03 07:40:47.481360 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:47.481323 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" event={"ID":"b0432b16-3bc9-42dc-84b6-b685dd7d0a88","Type":"ContainerStarted","Data":"dff620534429ea5d0ef7cb2befb4e058f9b587275b6dfe21bc0d9438690b05d5"} Nov 03 07:40:47.481765 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:47.481420 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:40:47.602889 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:47.602829 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" podStartSLOduration=2.7910740389999997 podStartE2EDuration="3.60281391s" totalImagesPullingTime="811.739871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:40:47.600396956 +0000 UTC m=+2502.792938707" watchObservedRunningTime="2026-11-03 07:40:47.60281391 +0000 UTC m=+2502.795355666" Nov 03 07:40:53.494946 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:40:53.494913 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:41:00.532319 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.532287 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp"] Nov 03 07:41:00.536653 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.536638 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.595914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.595881 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp"] Nov 03 07:41:00.640981 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.640925 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dcdb23\"/\"openshift-service-ca.crt\"" Nov 03 07:41:00.640981 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.640957 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dcdb23\"/\"default-dockercfg-bjcqp\"" Nov 03 07:41:00.640981 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.640958 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dcdb23\"/\"kube-root-ca.crt\"" Nov 03 07:41:00.641241 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.641012 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dcdb23\"/\"test-model-dcdb23-kserve-self-signed-certs\"" Nov 03 07:41:00.690457 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.690407 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-dshm\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.690677 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.690552 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-home\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.690677 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.690637 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547c2e42-d342-47b5-a9dd-d526027c7550-tls-certs\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.690927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.690733 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5c77\" (UniqueName: \"kubernetes.io/projected/547c2e42-d342-47b5-a9dd-d526027c7550-kube-api-access-r5c77\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.690927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.690796 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-tmp-dir\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.690927 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.690831 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-model-cache\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.711135 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.711100 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn"] Nov 03 07:41:00.721175 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.720867 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:00.759607 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.759574 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn"] Nov 03 07:41:00.776381 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.776354 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-eac348-serving-cert\"" Nov 03 07:41:00.792125 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792051 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-home\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.792281 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792145 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr46f\" (UniqueName: \"kubernetes.io/projected/18c3e73a-ca38-428f-b5ab-09b864ca5477-kube-api-access-pr46f\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:00.792281 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792179 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547c2e42-d342-47b5-a9dd-d526027c7550-tls-certs\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.792413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792294 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5c77\" (UniqueName: \"kubernetes.io/projected/547c2e42-d342-47b5-a9dd-d526027c7550-kube-api-access-r5c77\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.792413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792366 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-tmp-dir\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.792413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792400 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-model-cache\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.792583 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792453 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18c3e73a-ca38-428f-b5ab-09b864ca5477-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:00.792583 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792486 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-metrics-certs\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:00.792583 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792513 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-home\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.792583 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792526 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-dshm\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.792835 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792662 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-tmp-dir\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.792835 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792781 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-maas-api-tls\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:00.792952 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.792834 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-model-cache\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.794932 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.794909 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-dshm\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.794932 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.794924 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547c2e42-d342-47b5-a9dd-d526027c7550-tls-certs\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.856692 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.856662 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5c77\" (UniqueName: \"kubernetes.io/projected/547c2e42-d342-47b5-a9dd-d526027c7550-kube-api-access-r5c77\") pod \"test-model-dcdb23-kserve-6cbb8df78c-ldmhp\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:00.894172 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.894141 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18c3e73a-ca38-428f-b5ab-09b864ca5477-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:00.894316 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.894178 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-metrics-certs\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:00.894316 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.894249 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-maas-api-tls\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:00.894441 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.894365 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr46f\" (UniqueName: \"kubernetes.io/projected/18c3e73a-ca38-428f-b5ab-09b864ca5477-kube-api-access-pr46f\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:00.894441 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:41:00.894419 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-serving-cert: secret "maas-api-e2e-inf-b-eac348-serving-cert" not found Nov 03 07:41:00.894536 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:41:00.894487 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-maas-api-tls podName:18c3e73a-ca38-428f-b5ab-09b864ca5477 nodeName:}" failed. No retries permitted until 2026-11-03 07:41:01.394468864 +0000 UTC m=+2516.587010593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-maas-api-tls") pod "maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" (UID: "18c3e73a-ca38-428f-b5ab-09b864ca5477") : secret "maas-api-e2e-inf-b-eac348-serving-cert" not found Nov 03 07:41:00.894961 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.894863 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18c3e73a-ca38-428f-b5ab-09b864ca5477-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:00.896674 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.896645 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-metrics-certs\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:00.958062 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:00.958032 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr46f\" (UniqueName: \"kubernetes.io/projected/18c3e73a-ca38-428f-b5ab-09b864ca5477-kube-api-access-pr46f\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:01.149584 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:01.149546 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:01.400306 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:01.400216 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-maas-api-tls\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:01.402752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:01.402695 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-maas-api-tls\") pod \"maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:01.559477 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:01.559452 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp"] Nov 03 07:41:01.560678 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:41:01.560651 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod547c2e42_d342_47b5_a9dd_d526027c7550.slice/crio-c1595f4b2ea27fe6250c9e8cfb42ed570edaf06ba03a7dabc192494e63ea0fcc WatchSource:0}: Error finding container c1595f4b2ea27fe6250c9e8cfb42ed570edaf06ba03a7dabc192494e63ea0fcc: Status 404 returned error can't find the container with id c1595f4b2ea27fe6250c9e8cfb42ed570edaf06ba03a7dabc192494e63ea0fcc Nov 03 07:41:01.578759 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:01.578705 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" event={"ID":"547c2e42-d342-47b5-a9dd-d526027c7550","Type":"ContainerStarted","Data":"c1595f4b2ea27fe6250c9e8cfb42ed570edaf06ba03a7dabc192494e63ea0fcc"} Nov 03 07:41:01.637694 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:01.637673 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:02.069174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:02.069127 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn"] Nov 03 07:41:02.070857 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:41:02.070833 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c3e73a_ca38_428f_b5ab_09b864ca5477.slice/crio-4c8f26904de5ffa5a2589d890103f19fb65509da24de9cec6a682a9bbea60485 WatchSource:0}: Error finding container 4c8f26904de5ffa5a2589d890103f19fb65509da24de9cec6a682a9bbea60485: Status 404 returned error can't find the container with id 4c8f26904de5ffa5a2589d890103f19fb65509da24de9cec6a682a9bbea60485 Nov 03 07:41:02.588135 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:02.588096 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" event={"ID":"547c2e42-d342-47b5-a9dd-d526027c7550","Type":"ContainerStarted","Data":"3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0"} Nov 03 07:41:02.589442 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:02.589411 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" event={"ID":"18c3e73a-ca38-428f-b5ab-09b864ca5477","Type":"ContainerStarted","Data":"4c8f26904de5ffa5a2589d890103f19fb65509da24de9cec6a682a9bbea60485"} Nov 03 07:41:02.719769 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:02.719705 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" podStartSLOduration=2.719691257 podStartE2EDuration="2.719691257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:41:02.71758999 +0000 UTC m=+2517.910131744" watchObservedRunningTime="2026-11-03 07:41:02.719691257 +0000 UTC m=+2517.912233044" Nov 03 07:41:03.150232 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:03.150204 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:03.598921 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:03.598880 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" event={"ID":"18c3e73a-ca38-428f-b5ab-09b864ca5477","Type":"ContainerStarted","Data":"ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5"} Nov 03 07:41:03.723619 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:03.723566 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" podStartSLOduration=2.83823361 podStartE2EDuration="3.723552903s" totalImagesPullingTime="885.319293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:41:03.720313988 +0000 UTC m=+2518.912855731" watchObservedRunningTime="2026-11-03 07:41:03.723552903 +0000 UTC m=+2518.916094653" Nov 03 07:41:04.606110 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:04.606078 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:08.774344 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:41:08.774303 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:41:10.618255 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:10.618226 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:41:11.150882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:11.150849 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:11.163929 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:11.163904 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:11.659322 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:11.659297 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:41:19.394934 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:19.394898 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94"] Nov 03 07:41:26.428867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:26.428833 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw"] Nov 03 07:41:26.538643 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:26.538609 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59"] Nov 03 07:41:27.796208 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:27.796123 4409 generic.go:356] "Generic (PLEG): container finished" podID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerID="1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1" exitCode=0 Nov 03 07:41:27.796208 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:27.796198 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" event={"ID":"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed","Type":"ContainerDied","Data":"1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1"} Nov 03 07:41:27.796775 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:27.796649 4409 scope.go:122] "RemoveContainer" containerID="1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1" Nov 03 07:41:27.798362 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:27.798334 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:41:27.798433 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:27.798409 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:41:27.798484 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:27.798444 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:41:28.805312 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:28.805278 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" event={"ID":"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed","Type":"ContainerStarted","Data":"e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548"} Nov 03 07:41:29.524727 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:29.524686 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:41:29.525910 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:29.525880 4409 patch_prober.go:29] interesting pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 07:41:29.526042 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:29.525928 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 07:41:30.524830 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:30.524796 4409 patch_prober.go:29] interesting pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 07:41:30.525246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:30.524847 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 07:41:31.430133 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.430075 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" podUID="9f03bf4c-6138-4f5a-a3c7-739cb83219bb" containerName="payload-processing" containerID="cri-o://b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6" gracePeriod=25 Nov 03 07:41:31.525079 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.525049 4409 patch_prober.go:29] interesting pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 07:41:31.525436 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.525117 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 07:41:31.539225 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.539181 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" podUID="34efdb29-aee9-40f9-bb8e-92410e7b9e48" containerName="payload-pre-processing" containerID="cri-o://cfc21acad3b412e5eea8235e5cd5c4d3159f81a9bed5296f9b6f5c0c94bb8af0" gracePeriod=25 Nov 03 07:41:31.752218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.752182 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" Nov 03 07:41:31.818330 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.817365 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-plugins-config-volume\") pod \"9f03bf4c-6138-4f5a-a3c7-739cb83219bb\" (UID: \"9f03bf4c-6138-4f5a-a3c7-739cb83219bb\") " Nov 03 07:41:31.818330 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.817466 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-kube-api-access-gxfgc\" (UniqueName: \"kubernetes.io/projected/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-kube-api-access-gxfgc\") pod \"9f03bf4c-6138-4f5a-a3c7-739cb83219bb\" (UID: \"9f03bf4c-6138-4f5a-a3c7-739cb83219bb\") " Nov 03 07:41:31.818330 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.817788 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-plugins-config-volume" pod "9f03bf4c-6138-4f5a-a3c7-739cb83219bb" (UID: "9f03bf4c-6138-4f5a-a3c7-739cb83219bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:41:31.818330 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.818187 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:31.832266 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.832231 4409 generic.go:356] "Generic (PLEG): container finished" podID="9f03bf4c-6138-4f5a-a3c7-739cb83219bb" containerID="b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6" exitCode=0 Nov 03 07:41:31.832493 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.832316 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" Nov 03 07:41:31.832493 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.832439 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" event={"ID":"9f03bf4c-6138-4f5a-a3c7-739cb83219bb","Type":"ContainerDied","Data":"b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6"} Nov 03 07:41:31.832682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.832491 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw" event={"ID":"9f03bf4c-6138-4f5a-a3c7-739cb83219bb","Type":"ContainerDied","Data":"337e9cc704f1a118a8012deba72c27c55970d10fc9dbbf367c6f30053d9890de"} Nov 03 07:41:31.832682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.832524 4409 scope.go:122] "RemoveContainer" containerID="b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6" Nov 03 07:41:31.837263 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.836487 4409 generic.go:356] "Generic (PLEG): container finished" podID="34efdb29-aee9-40f9-bb8e-92410e7b9e48" containerID="cfc21acad3b412e5eea8235e5cd5c4d3159f81a9bed5296f9b6f5c0c94bb8af0" exitCode=0 Nov 03 07:41:31.837263 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.836664 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" event={"ID":"34efdb29-aee9-40f9-bb8e-92410e7b9e48","Type":"ContainerDied","Data":"cfc21acad3b412e5eea8235e5cd5c4d3159f81a9bed5296f9b6f5c0c94bb8af0"} Nov 03 07:41:31.896741 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.853064 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-kube-api-access-gxfgc" pod "9f03bf4c-6138-4f5a-a3c7-739cb83219bb" (UID: "9f03bf4c-6138-4f5a-a3c7-739cb83219bb"). InnerVolumeSpecName "kube-api-access-gxfgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:41:31.954746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.933423 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxfgc\" (UniqueName: \"kubernetes.io/projected/9f03bf4c-6138-4f5a-a3c7-739cb83219bb-kube-api-access-gxfgc\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:31.955031 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.954835 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:41:31.994154 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:31.993771 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" Nov 03 07:41:32.011059 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.011036 4409 scope.go:122] "RemoveContainer" containerID="b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6" Nov 03 07:41:32.017929 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:41:32.017882 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6\": container with ID starting with b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6 not found: ID does not exist" containerID="b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6" Nov 03 07:41:32.018069 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.017923 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6"} err="failed to get container status \"b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6\": rpc error: code = NotFound desc = could not find container \"b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6\": container with ID starting with b212c5916adb8158c56197e26eacf74bedc2d8980fc1cbdc517c9e786fbacfd6 not found: ID does not exist" Nov 03 07:41:32.135798 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.135761 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34efdb29-aee9-40f9-bb8e-92410e7b9e48-kube-api-access-mfg2d\" (UniqueName: \"kubernetes.io/projected/34efdb29-aee9-40f9-bb8e-92410e7b9e48-kube-api-access-mfg2d\") pod \"34efdb29-aee9-40f9-bb8e-92410e7b9e48\" (UID: \"34efdb29-aee9-40f9-bb8e-92410e7b9e48\") " Nov 03 07:41:32.135999 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.135875 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34efdb29-aee9-40f9-bb8e-92410e7b9e48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34efdb29-aee9-40f9-bb8e-92410e7b9e48-plugins-config-volume\") pod \"34efdb29-aee9-40f9-bb8e-92410e7b9e48\" (UID: \"34efdb29-aee9-40f9-bb8e-92410e7b9e48\") " Nov 03 07:41:32.136557 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.136506 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34efdb29-aee9-40f9-bb8e-92410e7b9e48-plugins-config-volume" pod "34efdb29-aee9-40f9-bb8e-92410e7b9e48" (UID: "34efdb29-aee9-40f9-bb8e-92410e7b9e48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:41:32.138744 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.138684 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34efdb29-aee9-40f9-bb8e-92410e7b9e48-kube-api-access-mfg2d" pod "34efdb29-aee9-40f9-bb8e-92410e7b9e48" (UID: "34efdb29-aee9-40f9-bb8e-92410e7b9e48"). InnerVolumeSpecName "kube-api-access-mfg2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:41:32.237216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.237187 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfg2d\" (UniqueName: \"kubernetes.io/projected/34efdb29-aee9-40f9-bb8e-92410e7b9e48-kube-api-access-mfg2d\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:32.237216 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.237213 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34efdb29-aee9-40f9-bb8e-92410e7b9e48-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:32.256810 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.256746 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw"] Nov 03 07:41:32.263079 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.263054 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9697f7-5b8bf9bf9c-pwrdw"] Nov 03 07:41:32.525246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.525168 4409 patch_prober.go:29] interesting pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 07:41:32.525246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.525219 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 07:41:32.846333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.846302 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" event={"ID":"34efdb29-aee9-40f9-bb8e-92410e7b9e48","Type":"ContainerDied","Data":"6d3d109251df5dd9943fe6a25e9ccd9efb9f7cbc7db092d0004f63b8081f0338"} Nov 03 07:41:32.846333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.846330 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59" Nov 03 07:41:32.846559 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.846382 4409 scope.go:122] "RemoveContainer" containerID="cfc21acad3b412e5eea8235e5cd5c4d3159f81a9bed5296f9b6f5c0c94bb8af0" Nov 03 07:41:32.913850 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:32.913815 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f03bf4c-6138-4f5a-a3c7-739cb83219bb" path="/var/lib/kubelet/pods/9f03bf4c-6138-4f5a-a3c7-739cb83219bb/volumes" Nov 03 07:41:33.017417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:33.017387 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59"] Nov 03 07:41:33.022991 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:33.022968 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-9697f7-5ff8d66455-c2c59"] Nov 03 07:41:33.525190 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:33.525153 4409 patch_prober.go:29] interesting pod/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 03 07:41:33.525380 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:33.525208 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 03 07:41:34.417305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.417261 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" containerID="cri-o://9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913" gracePeriod=45 Nov 03 07:41:34.529538 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.529399 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:41:34.530601 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.530582 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:41:34.692961 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.692938 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:41:34.767943 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.767916 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tmp-dir\") pod \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " Nov 03 07:41:34.768116 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.767963 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tls-certs\" (UniqueName: \"kubernetes.io/secret/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tls-certs\") pod \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " Nov 03 07:41:34.768116 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.767990 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-dshm\") pod \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " Nov 03 07:41:34.768116 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.768062 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-model-cache\") pod \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " Nov 03 07:41:34.768287 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.768156 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c21443-7e7e-46c0-835b-ad5cb444ddcb-kube-api-access-mf7l5\" (UniqueName: \"kubernetes.io/projected/12c21443-7e7e-46c0-835b-ad5cb444ddcb-kube-api-access-mf7l5\") pod \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " Nov 03 07:41:34.768287 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.768204 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tmp-dir" pod "12c21443-7e7e-46c0-835b-ad5cb444ddcb" (UID: "12c21443-7e7e-46c0-835b-ad5cb444ddcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.768287 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.768232 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-home\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-home\") pod \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\" (UID: \"12c21443-7e7e-46c0-835b-ad5cb444ddcb\") " Nov 03 07:41:34.768451 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.768356 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-model-cache" pod "12c21443-7e7e-46c0-835b-ad5cb444ddcb" (UID: "12c21443-7e7e-46c0-835b-ad5cb444ddcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.768547 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.768523 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-home" pod "12c21443-7e7e-46c0-835b-ad5cb444ddcb" (UID: "12c21443-7e7e-46c0-835b-ad5cb444ddcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.768759 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.768740 4409 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-home\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.768844 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.768765 4409 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tmp-dir\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.768844 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.768778 4409 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-model-cache\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.770318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.770297 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tls-certs" pod "12c21443-7e7e-46c0-835b-ad5cb444ddcb" (UID: "12c21443-7e7e-46c0-835b-ad5cb444ddcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:41:34.770318 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.770307 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-dshm" pod "12c21443-7e7e-46c0-835b-ad5cb444ddcb" (UID: "12c21443-7e7e-46c0-835b-ad5cb444ddcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:34.770412 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.770373 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c21443-7e7e-46c0-835b-ad5cb444ddcb-kube-api-access-mf7l5" pod "12c21443-7e7e-46c0-835b-ad5cb444ddcb" (UID: "12c21443-7e7e-46c0-835b-ad5cb444ddcb"). InnerVolumeSpecName "kube-api-access-mf7l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:41:34.867039 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.867004 4409 generic.go:356] "Generic (PLEG): container finished" podID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerID="9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913" exitCode=0 Nov 03 07:41:34.867231 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.867073 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" event={"ID":"12c21443-7e7e-46c0-835b-ad5cb444ddcb","Type":"ContainerDied","Data":"9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913"} Nov 03 07:41:34.867231 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.867119 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" event={"ID":"12c21443-7e7e-46c0-835b-ad5cb444ddcb","Type":"ContainerDied","Data":"19ee1f5d4e9c12788c95651859c14359c10782bb77cb3a71f6d911948680dae8"} Nov 03 07:41:34.867231 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.867142 4409 scope.go:122] "RemoveContainer" containerID="9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913" Nov 03 07:41:34.867231 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.867147 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94" Nov 03 07:41:34.870317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.870280 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf7l5\" (UniqueName: \"kubernetes.io/projected/12c21443-7e7e-46c0-835b-ad5cb444ddcb-kube-api-access-mf7l5\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.870317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.870311 4409 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c21443-7e7e-46c0-835b-ad5cb444ddcb-tls-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.870515 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.870327 4409 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c21443-7e7e-46c0-835b-ad5cb444ddcb-dshm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:34.879562 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.879541 4409 scope.go:122] "RemoveContainer" containerID="9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913" Nov 03 07:41:34.879896 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:41:34.879877 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913\": container with ID starting with 9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913 not found: ID does not exist" containerID="9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913" Nov 03 07:41:34.879983 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.879901 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913"} err="failed to get container status \"9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913\": rpc error: code = NotFound desc = could not find container \"9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913\": container with ID starting with 9ef87d225dd99a8e2d02636fc379346ff3b806d2ca12c8b0b4d13bc52192e913 not found: ID does not exist" Nov 03 07:41:34.914166 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:34.914137 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34efdb29-aee9-40f9-bb8e-92410e7b9e48" path="/var/lib/kubelet/pods/34efdb29-aee9-40f9-bb8e-92410e7b9e48/volumes" Nov 03 07:41:35.069597 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:35.069554 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94"] Nov 03 07:41:35.076866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:35.076827 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9697f7/distinct-w1-9697f7-kserve-69ddff45bd-7rr94"] Nov 03 07:41:36.480584 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:36.480545 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5"] Nov 03 07:41:36.480998 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:36.480819 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerName="istio-proxy" containerID="cri-o://0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71" gracePeriod=30 Nov 03 07:41:36.912977 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:36.912938 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" path="/var/lib/kubelet/pods/12c21443-7e7e-46c0-835b-ad5cb444ddcb/volumes" Nov 03 07:41:38.776879 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:41:38.776837 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:41:41.760220 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.760143 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:41:41.897635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.897601 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-socket\") pod \"7e54c4bd-d566-42ab-8613-67b296c03c18\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " Nov 03 07:41:41.897635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.897639 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-envoy\") pod \"7e54c4bd-d566-42ab-8613-67b296c03c18\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " Nov 03 07:41:41.897968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.897702 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-kube-api-access-vb2px\" (UniqueName: \"kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-kube-api-access-vb2px\") pod \"7e54c4bd-d566-42ab-8613-67b296c03c18\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " Nov 03 07:41:41.897968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.897772 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-certs\") pod \"7e54c4bd-d566-42ab-8613-67b296c03c18\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " Nov 03 07:41:41.897968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.897873 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-credential-socket\") pod \"7e54c4bd-d566-42ab-8613-67b296c03c18\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " Nov 03 07:41:41.897968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.897940 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e54c4bd-d566-42ab-8613-67b296c03c18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e54c4bd-d566-42ab-8613-67b296c03c18-istiod-ca-cert\") pod \"7e54c4bd-d566-42ab-8613-67b296c03c18\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " Nov 03 07:41:41.898182 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.897976 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-socket" pod "7e54c4bd-d566-42ab-8613-67b296c03c18" (UID: "7e54c4bd-d566-42ab-8613-67b296c03c18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:41.898246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.898000 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-data\") pod \"7e54c4bd-d566-42ab-8613-67b296c03c18\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " Nov 03 07:41:41.898297 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.898282 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-podinfo\") pod \"7e54c4bd-d566-42ab-8613-67b296c03c18\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " Nov 03 07:41:41.898361 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.898111 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-certs" pod "7e54c4bd-d566-42ab-8613-67b296c03c18" (UID: "7e54c4bd-d566-42ab-8613-67b296c03c18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:41.898361 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.898162 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-credential-socket" pod "7e54c4bd-d566-42ab-8613-67b296c03c18" (UID: "7e54c4bd-d566-42ab-8613-67b296c03c18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:41.898361 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.898242 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-data" pod "7e54c4bd-d566-42ab-8613-67b296c03c18" (UID: "7e54c4bd-d566-42ab-8613-67b296c03c18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:41.898479 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.898356 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e54c4bd-d566-42ab-8613-67b296c03c18-istiod-ca-cert" pod "7e54c4bd-d566-42ab-8613-67b296c03c18" (UID: "7e54c4bd-d566-42ab-8613-67b296c03c18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:41:41.898479 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.898452 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-token\" (UniqueName: \"kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-token\") pod \"7e54c4bd-d566-42ab-8613-67b296c03c18\" (UID: \"7e54c4bd-d566-42ab-8613-67b296c03c18\") " Nov 03 07:41:41.899284 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.899255 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.899423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.899286 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.899423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.899302 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.899423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.899318 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e54c4bd-d566-42ab-8613-67b296c03c18-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.899423 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.899334 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:41.900247 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.900209 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-envoy" pod "7e54c4bd-d566-42ab-8613-67b296c03c18" (UID: "7e54c4bd-d566-42ab-8613-67b296c03c18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:41:41.900508 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.900479 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-podinfo" pod "7e54c4bd-d566-42ab-8613-67b296c03c18" (UID: "7e54c4bd-d566-42ab-8613-67b296c03c18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:41:41.900508 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.900489 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-kube-api-access-vb2px" pod "7e54c4bd-d566-42ab-8613-67b296c03c18" (UID: "7e54c4bd-d566-42ab-8613-67b296c03c18"). InnerVolumeSpecName "kube-api-access-vb2px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:41:41.900833 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.900811 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-token" pod "7e54c4bd-d566-42ab-8613-67b296c03c18" (UID: "7e54c4bd-d566-42ab-8613-67b296c03c18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:41:41.918430 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.918401 4409 generic.go:356] "Generic (PLEG): container finished" podID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerID="0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71" exitCode=0 Nov 03 07:41:41.918591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.918451 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" Nov 03 07:41:41.918591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.918489 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" event={"ID":"7e54c4bd-d566-42ab-8613-67b296c03c18","Type":"ContainerDied","Data":"0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71"} Nov 03 07:41:41.918591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.918547 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5" event={"ID":"7e54c4bd-d566-42ab-8613-67b296c03c18","Type":"ContainerDied","Data":"2e78b0b2a61af65ca840fc9cfe5a5b75793a61ce4a908be6b36c2613c3254c5c"} Nov 03 07:41:41.918591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.918571 4409 scope.go:122] "RemoveContainer" containerID="0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71" Nov 03 07:41:41.930264 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.930238 4409 scope.go:122] "RemoveContainer" containerID="0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71" Nov 03 07:41:41.931471 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:41:41.930762 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71\": container with ID starting with 0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71 not found: ID does not exist" containerID="0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71" Nov 03 07:41:41.931471 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:41.930828 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71"} err="failed to get container status \"0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71\": rpc error: code = NotFound desc = could not find container \"0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71\": container with ID starting with 0f5382371024f0661faa608a6bec3aedc7441863ecc0520f428a9f3ae42e0a71 not found: ID does not exist" Nov 03 07:41:42.000921 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:42.000887 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb2px\" (UniqueName: \"kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-kube-api-access-vb2px\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:42.000921 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:42.000920 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:42.001151 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:42.000934 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:42.001151 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:42.000948 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e54c4bd-d566-42ab-8613-67b296c03c18-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:41:42.054629 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:42.054595 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5"] Nov 03 07:41:42.060204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:42.060170 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-3b41b9-openshift-default-84578dcc5f-9mlg5"] Nov 03 07:41:42.914187 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:41:42.914153 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" path="/var/lib/kubelet/pods/7e54c4bd-d566-42ab-8613-67b296c03c18/volumes" Nov 03 07:42:08.469557 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:08.469523 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74"] Nov 03 07:42:08.581158 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:08.581129 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv"] Nov 03 07:42:08.776874 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:42:08.776791 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:42:13.470642 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.470600 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" podUID="43579285-f289-4bdc-8e96-94c3798f3de1" containerName="payload-processing" containerID="cri-o://79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e" gracePeriod=25 Nov 03 07:42:13.581427 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.581393 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" podUID="93a5d08c-8abf-4e6f-8977-2485f249882c" containerName="payload-pre-processing" containerID="cri-o://842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd" gracePeriod=25 Nov 03 07:42:13.776475 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.776439 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" Nov 03 07:42:13.882693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.882673 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" Nov 03 07:42:13.889874 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.889845 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43579285-f289-4bdc-8e96-94c3798f3de1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43579285-f289-4bdc-8e96-94c3798f3de1-plugins-config-volume\") pod \"43579285-f289-4bdc-8e96-94c3798f3de1\" (UID: \"43579285-f289-4bdc-8e96-94c3798f3de1\") " Nov 03 07:42:13.890066 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.890053 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43579285-f289-4bdc-8e96-94c3798f3de1-kube-api-access-bhmnt\" (UniqueName: \"kubernetes.io/projected/43579285-f289-4bdc-8e96-94c3798f3de1-kube-api-access-bhmnt\") pod \"43579285-f289-4bdc-8e96-94c3798f3de1\" (UID: \"43579285-f289-4bdc-8e96-94c3798f3de1\") " Nov 03 07:42:13.890245 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.890218 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43579285-f289-4bdc-8e96-94c3798f3de1-plugins-config-volume" pod "43579285-f289-4bdc-8e96-94c3798f3de1" (UID: "43579285-f289-4bdc-8e96-94c3798f3de1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:42:13.890399 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.890385 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43579285-f289-4bdc-8e96-94c3798f3de1-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:13.892067 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.892048 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43579285-f289-4bdc-8e96-94c3798f3de1-kube-api-access-bhmnt" pod "43579285-f289-4bdc-8e96-94c3798f3de1" (UID: "43579285-f289-4bdc-8e96-94c3798f3de1"). InnerVolumeSpecName "kube-api-access-bhmnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:42:13.991504 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.991472 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a5d08c-8abf-4e6f-8977-2485f249882c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a5d08c-8abf-4e6f-8977-2485f249882c-plugins-config-volume\") pod \"93a5d08c-8abf-4e6f-8977-2485f249882c\" (UID: \"93a5d08c-8abf-4e6f-8977-2485f249882c\") " Nov 03 07:42:13.991684 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.991607 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a5d08c-8abf-4e6f-8977-2485f249882c-kube-api-access-gf8vg\" (UniqueName: \"kubernetes.io/projected/93a5d08c-8abf-4e6f-8977-2485f249882c-kube-api-access-gf8vg\") pod \"93a5d08c-8abf-4e6f-8977-2485f249882c\" (UID: \"93a5d08c-8abf-4e6f-8977-2485f249882c\") " Nov 03 07:42:13.991897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.991872 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a5d08c-8abf-4e6f-8977-2485f249882c-plugins-config-volume" pod "93a5d08c-8abf-4e6f-8977-2485f249882c" (UID: "93a5d08c-8abf-4e6f-8977-2485f249882c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:42:13.992062 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.992048 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhmnt\" (UniqueName: \"kubernetes.io/projected/43579285-f289-4bdc-8e96-94c3798f3de1-kube-api-access-bhmnt\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:13.992101 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.992065 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a5d08c-8abf-4e6f-8977-2485f249882c-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:13.993799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:13.993778 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a5d08c-8abf-4e6f-8977-2485f249882c-kube-api-access-gf8vg" pod "93a5d08c-8abf-4e6f-8977-2485f249882c" (UID: "93a5d08c-8abf-4e6f-8977-2485f249882c"). InnerVolumeSpecName "kube-api-access-gf8vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:42:14.093081 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.093050 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf8vg\" (UniqueName: \"kubernetes.io/projected/93a5d08c-8abf-4e6f-8977-2485f249882c-kube-api-access-gf8vg\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:42:14.176158 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.176126 4409 generic.go:356] "Generic (PLEG): container finished" podID="43579285-f289-4bdc-8e96-94c3798f3de1" containerID="79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e" exitCode=0 Nov 03 07:42:14.176333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.176178 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" Nov 03 07:42:14.176333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.176206 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" event={"ID":"43579285-f289-4bdc-8e96-94c3798f3de1","Type":"ContainerDied","Data":"79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e"} Nov 03 07:42:14.176333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.176253 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74" event={"ID":"43579285-f289-4bdc-8e96-94c3798f3de1","Type":"ContainerDied","Data":"0ab6e51549209b7faee540325f077ec3da0245ec77a758b78aac410e8ca701e5"} Nov 03 07:42:14.176333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.176275 4409 scope.go:122] "RemoveContainer" containerID="79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e" Nov 03 07:42:14.177864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.177683 4409 generic.go:356] "Generic (PLEG): container finished" podID="93a5d08c-8abf-4e6f-8977-2485f249882c" containerID="842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd" exitCode=0 Nov 03 07:42:14.177864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.177775 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" event={"ID":"93a5d08c-8abf-4e6f-8977-2485f249882c","Type":"ContainerDied","Data":"842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd"} Nov 03 07:42:14.177864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.177828 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" event={"ID":"93a5d08c-8abf-4e6f-8977-2485f249882c","Type":"ContainerDied","Data":"60c3541ff9aa5bcf726e7404498968a29bd639b8f15fa35855da43ee113020e8"} Nov 03 07:42:14.178220 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.178190 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv" Nov 03 07:42:14.187684 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.187668 4409 scope.go:122] "RemoveContainer" containerID="79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e" Nov 03 07:42:14.188002 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:42:14.187984 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e\": container with ID starting with 79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e not found: ID does not exist" containerID="79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e" Nov 03 07:42:14.188088 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.188011 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e"} err="failed to get container status \"79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e\": rpc error: code = NotFound desc = could not find container \"79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e\": container with ID starting with 79ecf07d123fe4fc251ea2a2add06702dc6c99bed8a3a2fa78bb8f7b0c6cf92e not found: ID does not exist" Nov 03 07:42:14.188088 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.188049 4409 scope.go:122] "RemoveContainer" containerID="842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd" Nov 03 07:42:14.197157 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.197139 4409 scope.go:122] "RemoveContainer" containerID="842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd" Nov 03 07:42:14.197428 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:42:14.197408 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd\": container with ID starting with 842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd not found: ID does not exist" containerID="842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd" Nov 03 07:42:14.197499 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.197433 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd"} err="failed to get container status \"842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd\": rpc error: code = NotFound desc = could not find container \"842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd\": container with ID starting with 842a627d5ed4a85a101a662b0d536e46132d16fab17c5ae22f9d2594a7e695dd not found: ID does not exist" Nov 03 07:42:14.345060 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.344992 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74"] Nov 03 07:42:14.350829 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.350807 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a8983d-855dc8669c-pgb74"] Nov 03 07:42:14.461895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.461867 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv"] Nov 03 07:42:14.467487 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.467464 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a8983d-6755cdc7c9-mkltv"] Nov 03 07:42:14.913205 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.913172 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43579285-f289-4bdc-8e96-94c3798f3de1" path="/var/lib/kubelet/pods/43579285-f289-4bdc-8e96-94c3798f3de1/volumes" Nov 03 07:42:14.913697 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:42:14.913534 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a5d08c-8abf-4e6f-8977-2485f249882c" path="/var/lib/kubelet/pods/93a5d08c-8abf-4e6f-8977-2485f249882c/volumes" Nov 03 07:42:38.778201 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:42:38.778154 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:43:08.774258 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:43:08.774223 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:43:38.775168 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:43:38.775128 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:43:38.793781 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:38.793746 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn"] Nov 03 07:43:38.794023 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:38.793997 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" podUID="6a5d1906-d406-4178-8be5-2af12fc35c35" containerName="maas-api" containerID="cri-o://97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618" gracePeriod=30 Nov 03 07:43:38.836579 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:38.836534 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" podUID="6a5d1906-d406-4178-8be5-2af12fc35c35" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.149:8443/health\": dial tcp 10.133.0.149:8443: connect: connection refused" Nov 03 07:43:39.102333 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.102304 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:43:39.250655 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.250612 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5d1906-d406-4178-8be5-2af12fc35c35-kube-api-access-5dk27\" (UniqueName: \"kubernetes.io/projected/6a5d1906-d406-4178-8be5-2af12fc35c35-kube-api-access-5dk27\") pod \"6a5d1906-d406-4178-8be5-2af12fc35c35\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " Nov 03 07:43:39.250655 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.250655 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-maas-api-tls\") pod \"6a5d1906-d406-4178-8be5-2af12fc35c35\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " Nov 03 07:43:39.250954 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.250688 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5d1906-d406-4178-8be5-2af12fc35c35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5d1906-d406-4178-8be5-2af12fc35c35-openshift-service-ca\") pod \"6a5d1906-d406-4178-8be5-2af12fc35c35\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " Nov 03 07:43:39.250954 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.250772 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-metrics-certs\") pod \"6a5d1906-d406-4178-8be5-2af12fc35c35\" (UID: \"6a5d1906-d406-4178-8be5-2af12fc35c35\") " Nov 03 07:43:39.251142 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.251114 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5d1906-d406-4178-8be5-2af12fc35c35-openshift-service-ca" pod "6a5d1906-d406-4178-8be5-2af12fc35c35" (UID: "6a5d1906-d406-4178-8be5-2af12fc35c35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:43:39.251679 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.251439 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5d1906-d406-4178-8be5-2af12fc35c35-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:39.253143 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.253105 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-maas-api-tls" pod "6a5d1906-d406-4178-8be5-2af12fc35c35" (UID: "6a5d1906-d406-4178-8be5-2af12fc35c35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:43:39.253274 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.253245 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5d1906-d406-4178-8be5-2af12fc35c35-kube-api-access-5dk27" pod "6a5d1906-d406-4178-8be5-2af12fc35c35" (UID: "6a5d1906-d406-4178-8be5-2af12fc35c35"). InnerVolumeSpecName "kube-api-access-5dk27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:43:39.253563 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.253532 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-metrics-certs" pod "6a5d1906-d406-4178-8be5-2af12fc35c35" (UID: "6a5d1906-d406-4178-8be5-2af12fc35c35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:43:39.361811 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.352873 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:39.362082 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.362053 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dk27\" (UniqueName: \"kubernetes.io/projected/6a5d1906-d406-4178-8be5-2af12fc35c35-kube-api-access-5dk27\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:39.362165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.362155 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5d1906-d406-4178-8be5-2af12fc35c35-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:39.707252 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.707163 4409 generic.go:356] "Generic (PLEG): container finished" podID="6a5d1906-d406-4178-8be5-2af12fc35c35" containerID="97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618" exitCode=0 Nov 03 07:43:39.707252 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.707215 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" Nov 03 07:43:39.707480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.707252 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" event={"ID":"6a5d1906-d406-4178-8be5-2af12fc35c35","Type":"ContainerDied","Data":"97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618"} Nov 03 07:43:39.707480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.707307 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn" event={"ID":"6a5d1906-d406-4178-8be5-2af12fc35c35","Type":"ContainerDied","Data":"1630fa4b5a5b9359e7dd9b3769a852fbe1f207a441540a88a56bcd91767b85d1"} Nov 03 07:43:39.707480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.707331 4409 scope.go:122] "RemoveContainer" containerID="97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618" Nov 03 07:43:39.718829 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.718808 4409 scope.go:122] "RemoveContainer" containerID="97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618" Nov 03 07:43:39.719144 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:43:39.719124 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618\": container with ID starting with 97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618 not found: ID does not exist" containerID="97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618" Nov 03 07:43:39.719196 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.719153 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618"} err="failed to get container status \"97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618\": rpc error: code = NotFound desc = could not find container \"97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618\": container with ID starting with 97cfee63292c627a569e1c687c1e6eb99462eb1cc04a721a1af0bde7ad24d618 not found: ID does not exist" Nov 03 07:43:39.889507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.889468 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn"] Nov 03 07:43:39.938459 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:39.938423 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fb4bc-bd95c9fcf-q29nn"] Nov 03 07:43:40.913833 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:40.913795 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5d1906-d406-4178-8be5-2af12fc35c35" path="/var/lib/kubelet/pods/6a5d1906-d406-4178-8be5-2af12fc35c35/volumes" Nov 03 07:43:48.466172 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:48.466138 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc"] Nov 03 07:43:48.466736 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:48.466408 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" containerID="cri-o://99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759" gracePeriod=30 Nov 03 07:43:49.691817 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:49.691787 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb"] Nov 03 07:43:49.692277 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:49.692004 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" containerID="cri-o://fbb21adff0057e9b93c38f3341d192294bce0319a1ba099ea2cddfec910cda82" gracePeriod=30 Nov 03 07:43:53.744315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.744257 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:43:53.829739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.829692 4409 generic.go:356] "Generic (PLEG): container finished" podID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerID="99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759" exitCode=0 Nov 03 07:43:53.829896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.829779 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" Nov 03 07:43:53.829896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.829782 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" event={"ID":"d86da6dc-a180-4dd3-86e0-b34c28b47c37","Type":"ContainerDied","Data":"99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759"} Nov 03 07:43:53.829896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.829860 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc" event={"ID":"d86da6dc-a180-4dd3-86e0-b34c28b47c37","Type":"ContainerDied","Data":"5b1f2a733efe3c53de8a06a6dae065b4ccf96a82302ab37b2701b507c06c037d"} Nov 03 07:43:53.829896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.829890 4409 scope.go:122] "RemoveContainer" containerID="99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759" Nov 03 07:43:53.834270 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.834244 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-token\" (UniqueName: \"kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-token\") pod \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " Nov 03 07:43:53.834401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.834298 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-certs\") pod \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " Nov 03 07:43:53.834401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.834327 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-data\") pod \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " Nov 03 07:43:53.834401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.834392 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-kube-api-access-8g667\" (UniqueName: \"kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-kube-api-access-8g667\") pod \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " Nov 03 07:43:53.834588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.834428 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-socket\") pod \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " Nov 03 07:43:53.834588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.834472 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-podinfo\") pod \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " Nov 03 07:43:53.834588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.834547 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istiod-ca-cert\") pod \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " Nov 03 07:43:53.834801 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.834637 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-envoy\") pod \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " Nov 03 07:43:53.834801 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.834725 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-credential-socket\") pod \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\" (UID: \"d86da6dc-a180-4dd3-86e0-b34c28b47c37\") " Nov 03 07:43:53.834923 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.834886 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-data" pod "d86da6dc-a180-4dd3-86e0-b34c28b47c37" (UID: "d86da6dc-a180-4dd3-86e0-b34c28b47c37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:43:53.835418 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.834996 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istiod-ca-cert" pod "d86da6dc-a180-4dd3-86e0-b34c28b47c37" (UID: "d86da6dc-a180-4dd3-86e0-b34c28b47c37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:43:53.835418 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.835245 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:53.835418 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.835212 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-credential-socket" pod "d86da6dc-a180-4dd3-86e0-b34c28b47c37" (UID: "d86da6dc-a180-4dd3-86e0-b34c28b47c37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:43:53.835418 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.835203 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-certs" pod "d86da6dc-a180-4dd3-86e0-b34c28b47c37" (UID: "d86da6dc-a180-4dd3-86e0-b34c28b47c37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:43:53.835418 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.835336 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-socket" pod "d86da6dc-a180-4dd3-86e0-b34c28b47c37" (UID: "d86da6dc-a180-4dd3-86e0-b34c28b47c37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:43:53.836897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.836867 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-kube-api-access-8g667" pod "d86da6dc-a180-4dd3-86e0-b34c28b47c37" (UID: "d86da6dc-a180-4dd3-86e0-b34c28b47c37"). InnerVolumeSpecName "kube-api-access-8g667". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:43:53.837003 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.836966 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-envoy" pod "d86da6dc-a180-4dd3-86e0-b34c28b47c37" (UID: "d86da6dc-a180-4dd3-86e0-b34c28b47c37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:43:53.837401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.837382 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-token" pod "d86da6dc-a180-4dd3-86e0-b34c28b47c37" (UID: "d86da6dc-a180-4dd3-86e0-b34c28b47c37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:43:53.837401 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.837383 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-podinfo" pod "d86da6dc-a180-4dd3-86e0-b34c28b47c37" (UID: "d86da6dc-a180-4dd3-86e0-b34c28b47c37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:43:53.840092 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.840075 4409 scope.go:122] "RemoveContainer" containerID="9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e" Nov 03 07:43:53.861032 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.861017 4409 scope.go:122] "RemoveContainer" containerID="99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759" Nov 03 07:43:53.861302 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:43:53.861286 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759\": container with ID starting with 99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759 not found: ID does not exist" containerID="99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759" Nov 03 07:43:53.861357 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.861311 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759"} err="failed to get container status \"99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759\": rpc error: code = NotFound desc = could not find container \"99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759\": container with ID starting with 99baf23475c035b79ec614420de45659ec13915aff0b394e5b5c3f1a17ebe759 not found: ID does not exist" Nov 03 07:43:53.861357 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.861325 4409 scope.go:122] "RemoveContainer" containerID="9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e" Nov 03 07:43:53.861543 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:43:53.861527 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e\": container with ID starting with 9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e not found: ID does not exist" containerID="9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e" Nov 03 07:43:53.861595 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.861549 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e"} err="failed to get container status \"9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e\": rpc error: code = NotFound desc = could not find container \"9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e\": container with ID starting with 9c0d5eeebde2de0fc0c8534136f0d40bd02e807ba7252edb11f1ba6bc314f39e not found: ID does not exist" Nov 03 07:43:53.937092 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.937054 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:53.937092 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.937088 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:53.937304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.937102 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:53.937304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.937117 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:53.937304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.937131 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g667\" (UniqueName: \"kubernetes.io/projected/d86da6dc-a180-4dd3-86e0-b34c28b47c37-kube-api-access-8g667\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:53.937304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.937144 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86da6dc-a180-4dd3-86e0-b34c28b47c37-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:53.937304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.937157 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:53.937304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:53.937170 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86da6dc-a180-4dd3-86e0-b34c28b47c37-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:54.256561 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:54.256532 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc"] Nov 03 07:43:54.264250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:54.264225 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a8983d-openshift-default-57b8f8f4c9-lnwpc"] Nov 03 07:43:54.839833 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:54.839799 4409 generic.go:356] "Generic (PLEG): container finished" podID="25689a41-3fdd-401f-a99c-d59679618916" containerID="fbb21adff0057e9b93c38f3341d192294bce0319a1ba099ea2cddfec910cda82" exitCode=0 Nov 03 07:43:54.840434 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:54.840407 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" event={"ID":"25689a41-3fdd-401f-a99c-d59679618916","Type":"ContainerDied","Data":"fbb21adff0057e9b93c38f3341d192294bce0319a1ba099ea2cddfec910cda82"} Nov 03 07:43:54.840562 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:54.840473 4409 scope.go:122] "RemoveContainer" containerID="9cfec25f7888e1c10dbad56265d7919ef0c2179bdadcf2f4ee02a531777cdb65" Nov 03 07:43:54.917544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:54.917509 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" path="/var/lib/kubelet/pods/d86da6dc-a180-4dd3-86e0-b34c28b47c37/volumes" Nov 03 07:43:54.966055 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:54.965998 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:43:55.050383 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050356 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-kube-api-access-9rn4g\" (UniqueName: \"kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-kube-api-access-9rn4g\") pod \"25689a41-3fdd-401f-a99c-d59679618916\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " Nov 03 07:43:55.050522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050415 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-data\") pod \"25689a41-3fdd-401f-a99c-d59679618916\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " Nov 03 07:43:55.050522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050452 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25689a41-3fdd-401f-a99c-d59679618916-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25689a41-3fdd-401f-a99c-d59679618916-istiod-ca-cert\") pod \"25689a41-3fdd-401f-a99c-d59679618916\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " Nov 03 07:43:55.050522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050482 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25689a41-3fdd-401f-a99c-d59679618916-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25689a41-3fdd-401f-a99c-d59679618916-istio-podinfo\") pod \"25689a41-3fdd-401f-a99c-d59679618916\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " Nov 03 07:43:55.050679 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050558 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-istio-token\" (UniqueName: \"kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-istio-token\") pod \"25689a41-3fdd-401f-a99c-d59679618916\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " Nov 03 07:43:55.050679 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050596 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-socket\") pod \"25689a41-3fdd-401f-a99c-d59679618916\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " Nov 03 07:43:55.050679 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050641 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-certs\") pod \"25689a41-3fdd-401f-a99c-d59679618916\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " Nov 03 07:43:55.050882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050688 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-credential-socket\") pod \"25689a41-3fdd-401f-a99c-d59679618916\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " Nov 03 07:43:55.050882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050701 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-data" pod "25689a41-3fdd-401f-a99c-d59679618916" (UID: "25689a41-3fdd-401f-a99c-d59679618916"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:43:55.050882 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050782 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-envoy\") pod \"25689a41-3fdd-401f-a99c-d59679618916\" (UID: \"25689a41-3fdd-401f-a99c-d59679618916\") " Nov 03 07:43:55.051058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050924 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-certs" pod "25689a41-3fdd-401f-a99c-d59679618916" (UID: "25689a41-3fdd-401f-a99c-d59679618916"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:43:55.051058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050944 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-credential-socket" pod "25689a41-3fdd-401f-a99c-d59679618916" (UID: "25689a41-3fdd-401f-a99c-d59679618916"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:43:55.051058 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.050962 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25689a41-3fdd-401f-a99c-d59679618916-istiod-ca-cert" pod "25689a41-3fdd-401f-a99c-d59679618916" (UID: "25689a41-3fdd-401f-a99c-d59679618916"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:43:55.051390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.051272 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:55.051390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.051293 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25689a41-3fdd-401f-a99c-d59679618916-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:55.051390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.051308 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:55.051390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.051322 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:55.051390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.051377 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-socket" pod "25689a41-3fdd-401f-a99c-d59679618916" (UID: "25689a41-3fdd-401f-a99c-d59679618916"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:43:55.052696 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.052675 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25689a41-3fdd-401f-a99c-d59679618916-istio-podinfo" pod "25689a41-3fdd-401f-a99c-d59679618916" (UID: "25689a41-3fdd-401f-a99c-d59679618916"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:43:55.052897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.052881 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-istio-token" pod "25689a41-3fdd-401f-a99c-d59679618916" (UID: "25689a41-3fdd-401f-a99c-d59679618916"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:43:55.053145 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.053124 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-kube-api-access-9rn4g" pod "25689a41-3fdd-401f-a99c-d59679618916" (UID: "25689a41-3fdd-401f-a99c-d59679618916"). InnerVolumeSpecName "kube-api-access-9rn4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:43:55.053256 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.053206 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-envoy" pod "25689a41-3fdd-401f-a99c-d59679618916" (UID: "25689a41-3fdd-401f-a99c-d59679618916"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:43:55.152785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.152761 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:55.152785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.152784 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:55.152968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.152794 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25689a41-3fdd-401f-a99c-d59679618916-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:55.152968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.152803 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rn4g\" (UniqueName: \"kubernetes.io/projected/25689a41-3fdd-401f-a99c-d59679618916-kube-api-access-9rn4g\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:55.152968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.152811 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25689a41-3fdd-401f-a99c-d59679618916-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:43:55.848845 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.848817 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" Nov 03 07:43:55.849270 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.848860 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb" event={"ID":"25689a41-3fdd-401f-a99c-d59679618916","Type":"ContainerDied","Data":"1ba8f88ec2cbc454c5e1aa48e60878c7c2fbc51fdeefd8b70c35fd74dd33408f"} Nov 03 07:43:55.849270 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:55.848921 4409 scope.go:122] "RemoveContainer" containerID="fbb21adff0057e9b93c38f3341d192294bce0319a1ba099ea2cddfec910cda82" Nov 03 07:43:56.018774 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:56.018745 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb"] Nov 03 07:43:56.024266 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:56.024248 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1fb4bc-openshift-default-5c9fc7df49-8cbgb"] Nov 03 07:43:56.912295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:43:56.912260 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25689a41-3fdd-401f-a99c-d59679618916" path="/var/lib/kubelet/pods/25689a41-3fdd-401f-a99c-d59679618916/volumes" Nov 03 07:44:03.454176 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:03.454136 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm"] Nov 03 07:44:03.571899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:03.571867 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s"] Nov 03 07:44:05.341813 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:05.341787 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:44:05.343617 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:05.343592 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:44:08.454488 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.454432 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" podUID="b01d19dc-7c01-4944-bf95-792f185c575c" containerName="payload-processing" containerID="cri-o://2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e" gracePeriod=25 Nov 03 07:44:08.573192 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.573148 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" podUID="c2810d11-55ad-45fb-8a49-4c5760fd60e8" containerName="payload-pre-processing" containerID="cri-o://368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b" gracePeriod=25 Nov 03 07:44:08.722054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.721974 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" Nov 03 07:44:08.783860 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:44:08.783826 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:44:08.812645 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.812620 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01d19dc-7c01-4944-bf95-792f185c575c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01d19dc-7c01-4944-bf95-792f185c575c-plugins-config-volume\") pod \"b01d19dc-7c01-4944-bf95-792f185c575c\" (UID: \"b01d19dc-7c01-4944-bf95-792f185c575c\") " Nov 03 07:44:08.812804 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.812765 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01d19dc-7c01-4944-bf95-792f185c575c-kube-api-access-nfmhs\" (UniqueName: \"kubernetes.io/projected/b01d19dc-7c01-4944-bf95-792f185c575c-kube-api-access-nfmhs\") pod \"b01d19dc-7c01-4944-bf95-792f185c575c\" (UID: \"b01d19dc-7c01-4944-bf95-792f185c575c\") " Nov 03 07:44:08.812975 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.812953 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01d19dc-7c01-4944-bf95-792f185c575c-plugins-config-volume" pod "b01d19dc-7c01-4944-bf95-792f185c575c" (UID: "b01d19dc-7c01-4944-bf95-792f185c575c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:44:08.813212 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.813195 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01d19dc-7c01-4944-bf95-792f185c575c-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:44:08.814887 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.814859 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01d19dc-7c01-4944-bf95-792f185c575c-kube-api-access-nfmhs" pod "b01d19dc-7c01-4944-bf95-792f185c575c" (UID: "b01d19dc-7c01-4944-bf95-792f185c575c"). InnerVolumeSpecName "kube-api-access-nfmhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:44:08.827885 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.827861 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" Nov 03 07:44:08.913605 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.913572 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2810d11-55ad-45fb-8a49-4c5760fd60e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2810d11-55ad-45fb-8a49-4c5760fd60e8-plugins-config-volume\") pod \"c2810d11-55ad-45fb-8a49-4c5760fd60e8\" (UID: \"c2810d11-55ad-45fb-8a49-4c5760fd60e8\") " Nov 03 07:44:08.913777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.913746 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2810d11-55ad-45fb-8a49-4c5760fd60e8-kube-api-access-t8sqz\" (UniqueName: \"kubernetes.io/projected/c2810d11-55ad-45fb-8a49-4c5760fd60e8-kube-api-access-t8sqz\") pod \"c2810d11-55ad-45fb-8a49-4c5760fd60e8\" (UID: \"c2810d11-55ad-45fb-8a49-4c5760fd60e8\") " Nov 03 07:44:08.913955 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.913928 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2810d11-55ad-45fb-8a49-4c5760fd60e8-plugins-config-volume" pod "c2810d11-55ad-45fb-8a49-4c5760fd60e8" (UID: "c2810d11-55ad-45fb-8a49-4c5760fd60e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:44:08.914287 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.914263 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2810d11-55ad-45fb-8a49-4c5760fd60e8-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:44:08.914383 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.914291 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfmhs\" (UniqueName: \"kubernetes.io/projected/b01d19dc-7c01-4944-bf95-792f185c575c-kube-api-access-nfmhs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:44:08.915894 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.915876 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2810d11-55ad-45fb-8a49-4c5760fd60e8-kube-api-access-t8sqz" pod "c2810d11-55ad-45fb-8a49-4c5760fd60e8" (UID: "c2810d11-55ad-45fb-8a49-4c5760fd60e8"). InnerVolumeSpecName "kube-api-access-t8sqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:44:08.945509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.945483 4409 generic.go:356] "Generic (PLEG): container finished" podID="c2810d11-55ad-45fb-8a49-4c5760fd60e8" containerID="368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b" exitCode=0 Nov 03 07:44:08.945626 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.945533 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" Nov 03 07:44:08.945626 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.945559 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" event={"ID":"c2810d11-55ad-45fb-8a49-4c5760fd60e8","Type":"ContainerDied","Data":"368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b"} Nov 03 07:44:08.945626 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.945606 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s" event={"ID":"c2810d11-55ad-45fb-8a49-4c5760fd60e8","Type":"ContainerDied","Data":"cfed9e5b8804a3baa977b0de9cca3cd1760b68664a238e5af03393ef7b7b47a5"} Nov 03 07:44:08.945843 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.945626 4409 scope.go:122] "RemoveContainer" containerID="368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b" Nov 03 07:44:08.947175 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.947114 4409 generic.go:356] "Generic (PLEG): container finished" podID="b01d19dc-7c01-4944-bf95-792f185c575c" containerID="2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e" exitCode=0 Nov 03 07:44:08.947175 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.947158 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" Nov 03 07:44:08.947387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.947158 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" event={"ID":"b01d19dc-7c01-4944-bf95-792f185c575c","Type":"ContainerDied","Data":"2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e"} Nov 03 07:44:08.947387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.947292 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm" event={"ID":"b01d19dc-7c01-4944-bf95-792f185c575c","Type":"ContainerDied","Data":"744ed7ddb03a54415740b0b043c9376d6db334b96ce7038ad61f6ac482d48d79"} Nov 03 07:44:08.956024 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.956010 4409 scope.go:122] "RemoveContainer" containerID="368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b" Nov 03 07:44:08.956293 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:44:08.956275 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b\": container with ID starting with 368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b not found: ID does not exist" containerID="368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b" Nov 03 07:44:08.956358 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.956299 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b"} err="failed to get container status \"368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b\": rpc error: code = NotFound desc = could not find container \"368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b\": container with ID starting with 368d6fb71170abd294a89fef2d71aaea0cb2a52ae6bb732251fe03d9cfe5432b not found: ID does not exist" Nov 03 07:44:08.956358 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.956315 4409 scope.go:122] "RemoveContainer" containerID="2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e" Nov 03 07:44:08.967174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.967155 4409 scope.go:122] "RemoveContainer" containerID="2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e" Nov 03 07:44:08.967453 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:44:08.967429 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e\": container with ID starting with 2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e not found: ID does not exist" containerID="2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e" Nov 03 07:44:08.967524 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:08.967474 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e"} err="failed to get container status \"2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e\": rpc error: code = NotFound desc = could not find container \"2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e\": container with ID starting with 2f8551139407994668d4f2479c22e6f54d368bb6f9ee31de8cc755cb43390f2e not found: ID does not exist" Nov 03 07:44:09.015121 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:09.015057 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8sqz\" (UniqueName: \"kubernetes.io/projected/c2810d11-55ad-45fb-8a49-4c5760fd60e8-kube-api-access-t8sqz\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:44:09.073375 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:09.073343 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s"] Nov 03 07:44:09.079668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:09.079646 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18b89b-7b94f6fdb7-zns5s"] Nov 03 07:44:09.192304 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:09.192272 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm"] Nov 03 07:44:09.196408 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:09.196388 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18b89b-74bcfbd577-5dfqm"] Nov 03 07:44:10.913221 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:10.913186 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01d19dc-7c01-4944-bf95-792f185c575c" path="/var/lib/kubelet/pods/b01d19dc-7c01-4944-bf95-792f185c575c/volumes" Nov 03 07:44:10.913687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:44:10.913563 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2810d11-55ad-45fb-8a49-4c5760fd60e8" path="/var/lib/kubelet/pods/c2810d11-55ad-45fb-8a49-4c5760fd60e8/volumes" Nov 03 07:44:38.775266 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:44:38.775213 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:45:04.488132 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:04.488090 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv"] Nov 03 07:45:04.488631 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:04.488322 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" containerID="cri-o://e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6" gracePeriod=30 Nov 03 07:45:08.773501 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:45:08.773458 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:45:09.749166 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.749139 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:45:09.930315 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.930279 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-certs\") pod \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " Nov 03 07:45:09.930837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.930360 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istiod-ca-cert\") pod \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " Nov 03 07:45:09.930837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.930412 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-credential-socket\") pod \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " Nov 03 07:45:09.930837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.930481 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-envoy\") pod \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " Nov 03 07:45:09.930837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.930527 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-token\" (UniqueName: \"kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-token\") pod \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " Nov 03 07:45:09.930837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.930555 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-podinfo\") pod \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " Nov 03 07:45:09.930837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.930574 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-certs" pod "8158bcb5-641e-40c7-9ac0-eb2d59f05235" (UID: "8158bcb5-641e-40c7-9ac0-eb2d59f05235"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:09.930837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.930603 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-kube-api-access-z6sc4\" (UniqueName: \"kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-kube-api-access-z6sc4\") pod \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " Nov 03 07:45:09.930837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.930701 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-socket\") pod \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " Nov 03 07:45:09.930837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.930786 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istiod-ca-cert" pod "8158bcb5-641e-40c7-9ac0-eb2d59f05235" (UID: "8158bcb5-641e-40c7-9ac0-eb2d59f05235"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:45:09.930837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.930788 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-data\") pod \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\" (UID: \"8158bcb5-641e-40c7-9ac0-eb2d59f05235\") " Nov 03 07:45:09.930837 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.930797 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-credential-socket" pod "8158bcb5-641e-40c7-9ac0-eb2d59f05235" (UID: "8158bcb5-641e-40c7-9ac0-eb2d59f05235"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:09.931620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.931001 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-data" pod "8158bcb5-641e-40c7-9ac0-eb2d59f05235" (UID: "8158bcb5-641e-40c7-9ac0-eb2d59f05235"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:09.931620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.931035 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-socket" pod "8158bcb5-641e-40c7-9ac0-eb2d59f05235" (UID: "8158bcb5-641e-40c7-9ac0-eb2d59f05235"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:09.931620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.931282 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:09.931620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.931303 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:09.931620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.931313 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:09.931620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.931322 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:09.931620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.931330 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:09.932864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.932836 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-token" pod "8158bcb5-641e-40c7-9ac0-eb2d59f05235" (UID: "8158bcb5-641e-40c7-9ac0-eb2d59f05235"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:45:09.933016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.932996 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-kube-api-access-z6sc4" pod "8158bcb5-641e-40c7-9ac0-eb2d59f05235" (UID: "8158bcb5-641e-40c7-9ac0-eb2d59f05235"). InnerVolumeSpecName "kube-api-access-z6sc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:45:09.933413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.933390 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-envoy" pod "8158bcb5-641e-40c7-9ac0-eb2d59f05235" (UID: "8158bcb5-641e-40c7-9ac0-eb2d59f05235"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:09.933413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:09.933394 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-podinfo" pod "8158bcb5-641e-40c7-9ac0-eb2d59f05235" (UID: "8158bcb5-641e-40c7-9ac0-eb2d59f05235"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:45:10.033097 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.033068 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6sc4\" (UniqueName: \"kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-kube-api-access-z6sc4\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:10.033097 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.033096 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:10.033097 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.033105 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:10.033324 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.033114 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8158bcb5-641e-40c7-9ac0-eb2d59f05235-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:10.296895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.296815 4409 generic.go:356] "Generic (PLEG): container finished" podID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerID="e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6" exitCode=0 Nov 03 07:45:10.296895 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.296881 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" Nov 03 07:45:10.297125 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.296906 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" event={"ID":"8158bcb5-641e-40c7-9ac0-eb2d59f05235","Type":"ContainerDied","Data":"e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6"} Nov 03 07:45:10.297125 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.296973 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv" event={"ID":"8158bcb5-641e-40c7-9ac0-eb2d59f05235","Type":"ContainerDied","Data":"9a8bc82cce734027678eb0d1e9e44f53134c98d7c1466cf396454086e224a06b"} Nov 03 07:45:10.297125 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.297003 4409 scope.go:122] "RemoveContainer" containerID="e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6" Nov 03 07:45:10.307904 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.307885 4409 scope.go:122] "RemoveContainer" containerID="c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6" Nov 03 07:45:10.317099 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.317078 4409 scope.go:122] "RemoveContainer" containerID="e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6" Nov 03 07:45:10.317340 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:45:10.317323 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6\": container with ID starting with e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6 not found: ID does not exist" containerID="e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6" Nov 03 07:45:10.317389 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.317347 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6"} err="failed to get container status \"e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6\": rpc error: code = NotFound desc = could not find container \"e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6\": container with ID starting with e99bb45cbca0d367137b173b358a6753987f389cc3994f54e4c35ff48c74f3d6 not found: ID does not exist" Nov 03 07:45:10.317389 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.317368 4409 scope.go:122] "RemoveContainer" containerID="c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6" Nov 03 07:45:10.317578 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:45:10.317564 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6\": container with ID starting with c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6 not found: ID does not exist" containerID="c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6" Nov 03 07:45:10.317617 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.317583 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6"} err="failed to get container status \"c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6\": rpc error: code = NotFound desc = could not find container \"c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6\": container with ID starting with c3893b640c820b4142299793befebec243ce316f2c402765763e1f39c6873df6 not found: ID does not exist" Nov 03 07:45:10.466007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.465982 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv"] Nov 03 07:45:10.515547 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.515520 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6606d6-openshift-default-86bfc64849-clgqv"] Nov 03 07:45:10.912904 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:10.912873 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" path="/var/lib/kubelet/pods/8158bcb5-641e-40c7-9ac0-eb2d59f05235/volumes" Nov 03 07:45:26.779035 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:26.779002 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j"] Nov 03 07:45:26.779415 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:26.779211 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" podUID="b1401ab1-a79a-43dc-9321-60679a32ca36" containerName="maas-api" containerID="cri-o://d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab" gracePeriod=30 Nov 03 07:45:27.089803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.089778 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:45:27.238586 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.238552 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1401ab1-a79a-43dc-9321-60679a32ca36-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1401ab1-a79a-43dc-9321-60679a32ca36-openshift-service-ca\") pod \"b1401ab1-a79a-43dc-9321-60679a32ca36\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " Nov 03 07:45:27.238804 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.238665 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-maas-api-tls\") pod \"b1401ab1-a79a-43dc-9321-60679a32ca36\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " Nov 03 07:45:27.238804 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.238773 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1401ab1-a79a-43dc-9321-60679a32ca36-kube-api-access-qd7gg\" (UniqueName: \"kubernetes.io/projected/b1401ab1-a79a-43dc-9321-60679a32ca36-kube-api-access-qd7gg\") pod \"b1401ab1-a79a-43dc-9321-60679a32ca36\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " Nov 03 07:45:27.238959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.238807 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-metrics-certs\") pod \"b1401ab1-a79a-43dc-9321-60679a32ca36\" (UID: \"b1401ab1-a79a-43dc-9321-60679a32ca36\") " Nov 03 07:45:27.239018 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.238963 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1401ab1-a79a-43dc-9321-60679a32ca36-openshift-service-ca" pod "b1401ab1-a79a-43dc-9321-60679a32ca36" (UID: "b1401ab1-a79a-43dc-9321-60679a32ca36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:45:27.239298 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.239278 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1401ab1-a79a-43dc-9321-60679a32ca36-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:27.241894 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.241867 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1401ab1-a79a-43dc-9321-60679a32ca36-kube-api-access-qd7gg" pod "b1401ab1-a79a-43dc-9321-60679a32ca36" (UID: "b1401ab1-a79a-43dc-9321-60679a32ca36"). InnerVolumeSpecName "kube-api-access-qd7gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:45:27.244071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.244022 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-metrics-certs" pod "b1401ab1-a79a-43dc-9321-60679a32ca36" (UID: "b1401ab1-a79a-43dc-9321-60679a32ca36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:45:27.245455 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.245430 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-maas-api-tls" pod "b1401ab1-a79a-43dc-9321-60679a32ca36" (UID: "b1401ab1-a79a-43dc-9321-60679a32ca36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:45:27.341593 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.341504 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:27.341887 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.341865 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd7gg\" (UniqueName: \"kubernetes.io/projected/b1401ab1-a79a-43dc-9321-60679a32ca36-kube-api-access-qd7gg\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:27.342018 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.342002 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1401ab1-a79a-43dc-9321-60679a32ca36-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:27.408675 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.408637 4409 generic.go:356] "Generic (PLEG): container finished" podID="b1401ab1-a79a-43dc-9321-60679a32ca36" containerID="d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab" exitCode=0 Nov 03 07:45:27.408897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.408690 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" Nov 03 07:45:27.408897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.408734 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" event={"ID":"b1401ab1-a79a-43dc-9321-60679a32ca36","Type":"ContainerDied","Data":"d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab"} Nov 03 07:45:27.408897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.408785 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j" event={"ID":"b1401ab1-a79a-43dc-9321-60679a32ca36","Type":"ContainerDied","Data":"cd83b70b09577817ed29a432a95dbfec9dd341937d111bb881f4569de8da9afe"} Nov 03 07:45:27.408897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.408807 4409 scope.go:122] "RemoveContainer" containerID="d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab" Nov 03 07:45:27.419599 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.419573 4409 scope.go:122] "RemoveContainer" containerID="d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab" Nov 03 07:45:27.419957 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:45:27.419936 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab\": container with ID starting with d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab not found: ID does not exist" containerID="d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab" Nov 03 07:45:27.420042 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.419964 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab"} err="failed to get container status \"d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab\": rpc error: code = NotFound desc = could not find container \"d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab\": container with ID starting with d95ec7da7a106f24f0b847412287dfdf044c02a6b9bd11bd215116181961aaab not found: ID does not exist" Nov 03 07:45:27.577521 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.577486 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j"] Nov 03 07:45:27.584381 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:27.584345 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18b89b-595795b66b-9fq9j"] Nov 03 07:45:28.912640 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:28.912598 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1401ab1-a79a-43dc-9321-60679a32ca36" path="/var/lib/kubelet/pods/b1401ab1-a79a-43dc-9321-60679a32ca36/volumes" Nov 03 07:45:33.291174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.291141 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b"] Nov 03 07:45:33.291682 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.291340 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" podUID="b0432b16-3bc9-42dc-84b6-b685dd7d0a88" containerName="maas-api" containerID="cri-o://dff620534429ea5d0ef7cb2befb4e058f9b587275b6dfe21bc0d9438690b05d5" gracePeriod=30 Nov 03 07:45:33.450080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.450050 4409 generic.go:356] "Generic (PLEG): container finished" podID="b0432b16-3bc9-42dc-84b6-b685dd7d0a88" containerID="dff620534429ea5d0ef7cb2befb4e058f9b587275b6dfe21bc0d9438690b05d5" exitCode=0 Nov 03 07:45:33.450228 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.450107 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" event={"ID":"b0432b16-3bc9-42dc-84b6-b685dd7d0a88","Type":"ContainerDied","Data":"dff620534429ea5d0ef7cb2befb4e058f9b587275b6dfe21bc0d9438690b05d5"} Nov 03 07:45:33.598921 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.598897 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:45:33.723354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.722146 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-kube-api-access-lg78v\" (UniqueName: \"kubernetes.io/projected/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-kube-api-access-lg78v\") pod \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " Nov 03 07:45:33.723354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.722218 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-openshift-service-ca\") pod \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " Nov 03 07:45:33.723354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.722332 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-maas-api-tls\") pod \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " Nov 03 07:45:33.723354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.722387 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-metrics-certs\") pod \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\" (UID: \"b0432b16-3bc9-42dc-84b6-b685dd7d0a88\") " Nov 03 07:45:33.729235 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.728523 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-metrics-certs" pod "b0432b16-3bc9-42dc-84b6-b685dd7d0a88" (UID: "b0432b16-3bc9-42dc-84b6-b685dd7d0a88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:45:33.729235 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.729112 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-openshift-service-ca" pod "b0432b16-3bc9-42dc-84b6-b685dd7d0a88" (UID: "b0432b16-3bc9-42dc-84b6-b685dd7d0a88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:45:33.730297 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.730268 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-kube-api-access-lg78v" pod "b0432b16-3bc9-42dc-84b6-b685dd7d0a88" (UID: "b0432b16-3bc9-42dc-84b6-b685dd7d0a88"). InnerVolumeSpecName "kube-api-access-lg78v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:45:33.730411 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.730359 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-maas-api-tls" pod "b0432b16-3bc9-42dc-84b6-b685dd7d0a88" (UID: "b0432b16-3bc9-42dc-84b6-b685dd7d0a88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:45:33.824071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.823996 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg78v\" (UniqueName: \"kubernetes.io/projected/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-kube-api-access-lg78v\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:33.824071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.824026 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:33.824071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.824044 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:33.824071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:33.824057 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0432b16-3bc9-42dc-84b6-b685dd7d0a88-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:34.457570 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:34.457542 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" Nov 03 07:45:34.458011 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:34.457542 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" event={"ID":"b0432b16-3bc9-42dc-84b6-b685dd7d0a88","Type":"ContainerDied","Data":"de48b6c02641bd79714e8400df31ec4ed4715e93040afe6f092af761cd1aa340"} Nov 03 07:45:34.458011 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:34.457737 4409 scope.go:122] "RemoveContainer" containerID="dff620534429ea5d0ef7cb2befb4e058f9b587275b6dfe21bc0d9438690b05d5" Nov 03 07:45:34.627094 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:34.627067 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b"] Nov 03 07:45:34.676835 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:34.676806 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b"] Nov 03 07:45:34.913041 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:34.913013 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0432b16-3bc9-42dc-84b6-b685dd7d0a88" path="/var/lib/kubelet/pods/b0432b16-3bc9-42dc-84b6-b685dd7d0a88/volumes" Nov 03 07:45:36.490179 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:36.490142 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-44c08f-65b69946f5-mcv7b" podUID="b0432b16-3bc9-42dc-84b6-b685dd7d0a88" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.155:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 07:45:38.775722 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:45:38.775673 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:45:43.148283 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:43.148248 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg"] Nov 03 07:45:43.203761 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:43.199079 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw"] Nov 03 07:45:49.404890 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:49.404852 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf"] Nov 03 07:45:58.184633 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.184518 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" containerID="cri-o://74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae" gracePeriod=45 Nov 03 07:45:58.227222 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.227186 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" containerID="cri-o://c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e" gracePeriod=45 Nov 03 07:45:58.493076 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.493051 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:45:58.525161 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.525131 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:45:58.563207 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.563186 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-home\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-home\") pod \"27976e39-e6a3-4269-966b-67cca0cf2d9e\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " Nov 03 07:45:58.563332 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.563319 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27976e39-e6a3-4269-966b-67cca0cf2d9e-tls-certs\" (UniqueName: \"kubernetes.io/secret/27976e39-e6a3-4269-966b-67cca0cf2d9e-tls-certs\") pod \"27976e39-e6a3-4269-966b-67cca0cf2d9e\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " Nov 03 07:45:58.563379 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.563363 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-tmp-dir\") pod \"27976e39-e6a3-4269-966b-67cca0cf2d9e\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " Nov 03 07:45:58.563415 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.563391 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-home" pod "27976e39-e6a3-4269-966b-67cca0cf2d9e" (UID: "27976e39-e6a3-4269-966b-67cca0cf2d9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:58.563415 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.563403 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-dshm\") pod \"27976e39-e6a3-4269-966b-67cca0cf2d9e\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " Nov 03 07:45:58.563488 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.563436 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27976e39-e6a3-4269-966b-67cca0cf2d9e-kube-api-access-5hgp9\" (UniqueName: \"kubernetes.io/projected/27976e39-e6a3-4269-966b-67cca0cf2d9e-kube-api-access-5hgp9\") pod \"27976e39-e6a3-4269-966b-67cca0cf2d9e\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " Nov 03 07:45:58.563488 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.563468 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-model-cache\") pod \"27976e39-e6a3-4269-966b-67cca0cf2d9e\" (UID: \"27976e39-e6a3-4269-966b-67cca0cf2d9e\") " Nov 03 07:45:58.563606 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.563524 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-tmp-dir" pod "27976e39-e6a3-4269-966b-67cca0cf2d9e" (UID: "27976e39-e6a3-4269-966b-67cca0cf2d9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:58.563781 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.563757 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-model-cache" pod "27976e39-e6a3-4269-966b-67cca0cf2d9e" (UID: "27976e39-e6a3-4269-966b-67cca0cf2d9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:58.563905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.563875 4409 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-model-cache\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.563905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.563894 4409 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-home\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.564002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.563906 4409 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-tmp-dir\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.565663 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.565638 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-dshm" pod "27976e39-e6a3-4269-966b-67cca0cf2d9e" (UID: "27976e39-e6a3-4269-966b-67cca0cf2d9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:58.565791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.565731 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27976e39-e6a3-4269-966b-67cca0cf2d9e-kube-api-access-5hgp9" pod "27976e39-e6a3-4269-966b-67cca0cf2d9e" (UID: "27976e39-e6a3-4269-966b-67cca0cf2d9e"). InnerVolumeSpecName "kube-api-access-5hgp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:45:58.565791 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.565743 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27976e39-e6a3-4269-966b-67cca0cf2d9e-tls-certs" pod "27976e39-e6a3-4269-966b-67cca0cf2d9e" (UID: "27976e39-e6a3-4269-966b-67cca0cf2d9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:45:58.617387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.617223 4409 generic.go:356] "Generic (PLEG): container finished" podID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerID="74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae" exitCode=0 Nov 03 07:45:58.617387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.617269 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" Nov 03 07:45:58.617387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.617267 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" event={"ID":"27976e39-e6a3-4269-966b-67cca0cf2d9e","Type":"ContainerDied","Data":"74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae"} Nov 03 07:45:58.617387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.617329 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg" event={"ID":"27976e39-e6a3-4269-966b-67cca0cf2d9e","Type":"ContainerDied","Data":"16fcce7a26a7d74549595be428d393006aa1dc3b0997627442156852ed22133f"} Nov 03 07:45:58.617387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.617353 4409 scope.go:122] "RemoveContainer" containerID="74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae" Nov 03 07:45:58.619868 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.619849 4409 generic.go:356] "Generic (PLEG): container finished" podID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerID="c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e" exitCode=0 Nov 03 07:45:58.620019 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.619887 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" event={"ID":"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c","Type":"ContainerDied","Data":"c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e"} Nov 03 07:45:58.620157 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.620134 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" event={"ID":"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c","Type":"ContainerDied","Data":"d3fcad00e31d55ea41bd8f63658b4c893b91843e4b41ec8c09ab8153853ffc50"} Nov 03 07:45:58.620302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.619907 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw" Nov 03 07:45:58.630100 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.629901 4409 scope.go:122] "RemoveContainer" containerID="74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae" Nov 03 07:45:58.630378 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:45:58.630255 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae\": container with ID starting with 74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae not found: ID does not exist" containerID="74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae" Nov 03 07:45:58.630378 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.630289 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae"} err="failed to get container status \"74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae\": rpc error: code = NotFound desc = could not find container \"74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae\": container with ID starting with 74b65dc3c730896956305b5c0e9c5a65cc745a38a75182bfef027964700a50ae not found: ID does not exist" Nov 03 07:45:58.630378 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.630311 4409 scope.go:122] "RemoveContainer" containerID="c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e" Nov 03 07:45:58.643204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.643174 4409 scope.go:122] "RemoveContainer" containerID="c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e" Nov 03 07:45:58.643549 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:45:58.643524 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e\": container with ID starting with c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e not found: ID does not exist" containerID="c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e" Nov 03 07:45:58.643632 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.643559 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e"} err="failed to get container status \"c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e\": rpc error: code = NotFound desc = could not find container \"c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e\": container with ID starting with c55ff9e3e4e9f88fffa709484d9ecc28315ee5d81607d9b122a389261b3aea0e not found: ID does not exist" Nov 03 07:45:58.665071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.665044 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-kube-api-access-x8j2w\" (UniqueName: \"kubernetes.io/projected/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-kube-api-access-x8j2w\") pod \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " Nov 03 07:45:58.665218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.665126 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tmp-dir\") pod \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " Nov 03 07:45:58.665218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.665160 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-home\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-home\") pod \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " Nov 03 07:45:58.665218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.665214 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-dshm\") pod \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " Nov 03 07:45:58.665438 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.665329 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tls-certs\" (UniqueName: \"kubernetes.io/secret/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tls-certs\") pod \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " Nov 03 07:45:58.665438 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.665350 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-model-cache\") pod \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\" (UID: \"78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c\") " Nov 03 07:45:58.665560 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.665432 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tmp-dir" pod "78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" (UID: "78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:58.665855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.665726 4409 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27976e39-e6a3-4269-966b-67cca0cf2d9e-dshm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.665855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.665742 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hgp9\" (UniqueName: \"kubernetes.io/projected/27976e39-e6a3-4269-966b-67cca0cf2d9e-kube-api-access-5hgp9\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.665855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.665751 4409 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tmp-dir\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.665855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.665759 4409 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27976e39-e6a3-4269-966b-67cca0cf2d9e-tls-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.666210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.665966 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-model-cache" pod "78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" (UID: "78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:58.666517 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.666493 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-home" pod "78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" (UID: "78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:58.667917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.667892 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-kube-api-access-x8j2w" pod "78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" (UID: "78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c"). InnerVolumeSpecName "kube-api-access-x8j2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:45:58.668210 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.668188 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-dshm" pod "78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" (UID: "78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:45:58.669385 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.669366 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tls-certs" pod "78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" (UID: "78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:45:58.766951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.766823 4409 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-tls-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.766951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.766848 4409 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-model-cache\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.766951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.766859 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8j2w\" (UniqueName: \"kubernetes.io/projected/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-kube-api-access-x8j2w\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.766951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.766874 4409 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-home\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.766951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.766887 4409 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c-dshm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:45:58.846248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.846217 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg"] Nov 03 07:45:58.851892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.851849 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/distinct-w3-1f2e7e-kserve-799769bfcb-56twg"] Nov 03 07:45:58.912554 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:58.912527 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" path="/var/lib/kubelet/pods/27976e39-e6a3-4269-966b-67cca0cf2d9e/volumes" Nov 03 07:45:59.036506 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:59.036441 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw"] Nov 03 07:45:59.042349 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:45:59.042326 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1f2e7e/premium-w3-1f2e7e-kserve-b658b5565-zl8cw"] Nov 03 07:46:00.912945 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:00.912912 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" path="/var/lib/kubelet/pods/78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c/volumes" Nov 03 07:46:04.425892 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.425850 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" containerID="cri-o://f2fe42ea6c3d5a18f25672e09d72437b6f9aaac5715a1e0f32d3f18f1219354f" gracePeriod=45 Nov 03 07:46:04.659914 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.659885 4409 generic.go:356] "Generic (PLEG): container finished" podID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerID="f2fe42ea6c3d5a18f25672e09d72437b6f9aaac5715a1e0f32d3f18f1219354f" exitCode=0 Nov 03 07:46:04.660067 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.659967 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" event={"ID":"cf0fec39-da43-4132-af60-9948d7f78cf3","Type":"ContainerDied","Data":"f2fe42ea6c3d5a18f25672e09d72437b6f9aaac5715a1e0f32d3f18f1219354f"} Nov 03 07:46:04.680978 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.680926 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:46:04.833208 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.833174 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-home\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-home\") pod \"cf0fec39-da43-4132-af60-9948d7f78cf3\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " Nov 03 07:46:04.833398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.833238 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0fec39-da43-4132-af60-9948d7f78cf3-kube-api-access-lsjg6\" (UniqueName: \"kubernetes.io/projected/cf0fec39-da43-4132-af60-9948d7f78cf3-kube-api-access-lsjg6\") pod \"cf0fec39-da43-4132-af60-9948d7f78cf3\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " Nov 03 07:46:04.833398 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.833374 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-model-cache\") pod \"cf0fec39-da43-4132-af60-9948d7f78cf3\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " Nov 03 07:46:04.833525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.833461 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-home" pod "cf0fec39-da43-4132-af60-9948d7f78cf3" (UID: "cf0fec39-da43-4132-af60-9948d7f78cf3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:04.833525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.833488 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf0fec39-da43-4132-af60-9948d7f78cf3-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf0fec39-da43-4132-af60-9948d7f78cf3-tls-certs\") pod \"cf0fec39-da43-4132-af60-9948d7f78cf3\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " Nov 03 07:46:04.833638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.833535 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-tmp-dir\") pod \"cf0fec39-da43-4132-af60-9948d7f78cf3\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " Nov 03 07:46:04.833638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.833563 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-dshm\") pod \"cf0fec39-da43-4132-af60-9948d7f78cf3\" (UID: \"cf0fec39-da43-4132-af60-9948d7f78cf3\") " Nov 03 07:46:04.833881 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.833844 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-tmp-dir" pod "cf0fec39-da43-4132-af60-9948d7f78cf3" (UID: "cf0fec39-da43-4132-af60-9948d7f78cf3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:04.834069 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.834050 4409 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-tmp-dir\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:04.834143 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.834073 4409 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-home\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:04.834143 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.834122 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-model-cache" pod "cf0fec39-da43-4132-af60-9948d7f78cf3" (UID: "cf0fec39-da43-4132-af60-9948d7f78cf3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:04.836759 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.836236 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0fec39-da43-4132-af60-9948d7f78cf3-kube-api-access-lsjg6" pod "cf0fec39-da43-4132-af60-9948d7f78cf3" (UID: "cf0fec39-da43-4132-af60-9948d7f78cf3"). InnerVolumeSpecName "kube-api-access-lsjg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:46:04.836759 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.836329 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-dshm" pod "cf0fec39-da43-4132-af60-9948d7f78cf3" (UID: "cf0fec39-da43-4132-af60-9948d7f78cf3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:04.836759 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.836425 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf0fec39-da43-4132-af60-9948d7f78cf3-tls-certs" pod "cf0fec39-da43-4132-af60-9948d7f78cf3" (UID: "cf0fec39-da43-4132-af60-9948d7f78cf3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:46:04.935307 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.935247 4409 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf0fec39-da43-4132-af60-9948d7f78cf3-tls-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:04.935307 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.935271 4409 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-dshm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:04.935307 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.935281 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsjg6\" (UniqueName: \"kubernetes.io/projected/cf0fec39-da43-4132-af60-9948d7f78cf3-kube-api-access-lsjg6\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:04.935307 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:04.935290 4409 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf0fec39-da43-4132-af60-9948d7f78cf3-model-cache\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:05.666883 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:05.666851 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" event={"ID":"cf0fec39-da43-4132-af60-9948d7f78cf3","Type":"ContainerDied","Data":"eb3e427f11e7602e13e7893ca10805aba730745d81770fedfbe76ef66314b13d"} Nov 03 07:46:05.667317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:05.666899 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf" Nov 03 07:46:05.667317 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:05.666903 4409 scope.go:122] "RemoveContainer" containerID="f2fe42ea6c3d5a18f25672e09d72437b6f9aaac5715a1e0f32d3f18f1219354f" Nov 03 07:46:05.834006 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:05.833974 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf"] Nov 03 07:46:05.883274 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:05.883243 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-669d6b/distinct-2-w2-669d6b-kserve-5b95879b69-5fwzf"] Nov 03 07:46:06.913078 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:06.913047 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" path="/var/lib/kubelet/pods/cf0fec39-da43-4132-af60-9948d7f78cf3/volumes" Nov 03 07:46:08.773811 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:46:08.773773 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:46:12.825686 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:12.825652 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp"] Nov 03 07:46:24.891869 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:24.891835 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9"] Nov 03 07:46:25.001680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:25.001639 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7"] Nov 03 07:46:27.847300 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:27.847230 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" podUID="547c2e42-d342-47b5-a9dd-d526027c7550" containerName="main" containerID="cri-o://3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0" gracePeriod=45 Nov 03 07:46:28.110904 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.110838 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:46:28.267917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.267885 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-home\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-home\") pod \"547c2e42-d342-47b5-a9dd-d526027c7550\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " Nov 03 07:46:28.268127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.267946 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-model-cache\") pod \"547c2e42-d342-47b5-a9dd-d526027c7550\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " Nov 03 07:46:28.268127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.268040 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-dshm\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-dshm\") pod \"547c2e42-d342-47b5-a9dd-d526027c7550\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " Nov 03 07:46:28.268127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.268066 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547c2e42-d342-47b5-a9dd-d526027c7550-kube-api-access-r5c77\" (UniqueName: \"kubernetes.io/projected/547c2e42-d342-47b5-a9dd-d526027c7550-kube-api-access-r5c77\") pod \"547c2e42-d342-47b5-a9dd-d526027c7550\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " Nov 03 07:46:28.268305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.268130 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547c2e42-d342-47b5-a9dd-d526027c7550-tls-certs\" (UniqueName: \"kubernetes.io/secret/547c2e42-d342-47b5-a9dd-d526027c7550-tls-certs\") pod \"547c2e42-d342-47b5-a9dd-d526027c7550\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " Nov 03 07:46:28.268305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.268209 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-tmp-dir\") pod \"547c2e42-d342-47b5-a9dd-d526027c7550\" (UID: \"547c2e42-d342-47b5-a9dd-d526027c7550\") " Nov 03 07:46:28.268305 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.268220 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-home" pod "547c2e42-d342-47b5-a9dd-d526027c7550" (UID: "547c2e42-d342-47b5-a9dd-d526027c7550"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:28.268462 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.268292 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-model-cache" pod "547c2e42-d342-47b5-a9dd-d526027c7550" (UID: "547c2e42-d342-47b5-a9dd-d526027c7550"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:28.268685 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.268640 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-tmp-dir" pod "547c2e42-d342-47b5-a9dd-d526027c7550" (UID: "547c2e42-d342-47b5-a9dd-d526027c7550"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:28.268685 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.268658 4409 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-home\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:28.268685 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.268679 4409 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-model-cache\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:28.270764 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.270697 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547c2e42-d342-47b5-a9dd-d526027c7550-tls-certs" pod "547c2e42-d342-47b5-a9dd-d526027c7550" (UID: "547c2e42-d342-47b5-a9dd-d526027c7550"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:46:28.271290 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.271268 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547c2e42-d342-47b5-a9dd-d526027c7550-kube-api-access-r5c77" pod "547c2e42-d342-47b5-a9dd-d526027c7550" (UID: "547c2e42-d342-47b5-a9dd-d526027c7550"). InnerVolumeSpecName "kube-api-access-r5c77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:46:28.271819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.271800 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-dshm" pod "547c2e42-d342-47b5-a9dd-d526027c7550" (UID: "547c2e42-d342-47b5-a9dd-d526027c7550"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:28.369544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.369454 4409 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-dshm\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:28.369544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.369487 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5c77\" (UniqueName: \"kubernetes.io/projected/547c2e42-d342-47b5-a9dd-d526027c7550-kube-api-access-r5c77\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:28.369544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.369499 4409 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547c2e42-d342-47b5-a9dd-d526027c7550-tls-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:28.369544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.369509 4409 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547c2e42-d342-47b5-a9dd-d526027c7550-tmp-dir\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:28.847629 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.847595 4409 generic.go:356] "Generic (PLEG): container finished" podID="547c2e42-d342-47b5-a9dd-d526027c7550" containerID="3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0" exitCode=0 Nov 03 07:46:28.848202 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.847637 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" Nov 03 07:46:28.848202 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.847659 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" event={"ID":"547c2e42-d342-47b5-a9dd-d526027c7550","Type":"ContainerDied","Data":"3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0"} Nov 03 07:46:28.848202 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.847701 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp" event={"ID":"547c2e42-d342-47b5-a9dd-d526027c7550","Type":"ContainerDied","Data":"c1595f4b2ea27fe6250c9e8cfb42ed570edaf06ba03a7dabc192494e63ea0fcc"} Nov 03 07:46:28.848202 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.847746 4409 scope.go:122] "RemoveContainer" containerID="3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0" Nov 03 07:46:28.858010 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.857813 4409 scope.go:122] "RemoveContainer" containerID="3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0" Nov 03 07:46:28.858135 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:46:28.858116 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0\": container with ID starting with 3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0 not found: ID does not exist" containerID="3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0" Nov 03 07:46:28.858194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:28.858147 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0"} err="failed to get container status \"3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0\": rpc error: code = NotFound desc = could not find container \"3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0\": container with ID starting with 3a2483ea2fe941ceeecc9532d1346fc97ef9a9df4b9d5663cba603a4c36db2f0 not found: ID does not exist" Nov 03 07:46:29.037848 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:29.037805 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp"] Nov 03 07:46:29.066417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:29.066379 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-dcdb23/test-model-dcdb23-kserve-6cbb8df78c-ldmhp"] Nov 03 07:46:29.892636 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:29.892580 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" podUID="dff76af4-5e55-4cd1-8341-c4d674fad0fa" containerName="payload-processing" containerID="cri-o://38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2" gracePeriod=25 Nov 03 07:46:30.002706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.002651 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" podUID="34261966-3f63-47b2-9bff-e687beb22e3e" containerName="payload-pre-processing" containerID="cri-o://3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02" gracePeriod=25 Nov 03 07:46:30.142594 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.142570 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" Nov 03 07:46:30.290554 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.290481 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff76af4-5e55-4cd1-8341-c4d674fad0fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff76af4-5e55-4cd1-8341-c4d674fad0fa-plugins-config-volume\") pod \"dff76af4-5e55-4cd1-8341-c4d674fad0fa\" (UID: \"dff76af4-5e55-4cd1-8341-c4d674fad0fa\") " Nov 03 07:46:30.290695 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.290602 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff76af4-5e55-4cd1-8341-c4d674fad0fa-kube-api-access-vzwsk\" (UniqueName: \"kubernetes.io/projected/dff76af4-5e55-4cd1-8341-c4d674fad0fa-kube-api-access-vzwsk\") pod \"dff76af4-5e55-4cd1-8341-c4d674fad0fa\" (UID: \"dff76af4-5e55-4cd1-8341-c4d674fad0fa\") " Nov 03 07:46:30.290950 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.290924 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff76af4-5e55-4cd1-8341-c4d674fad0fa-plugins-config-volume" pod "dff76af4-5e55-4cd1-8341-c4d674fad0fa" (UID: "dff76af4-5e55-4cd1-8341-c4d674fad0fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:46:30.291162 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.291138 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff76af4-5e55-4cd1-8341-c4d674fad0fa-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:30.292997 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.292965 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff76af4-5e55-4cd1-8341-c4d674fad0fa-kube-api-access-vzwsk" pod "dff76af4-5e55-4cd1-8341-c4d674fad0fa" (UID: "dff76af4-5e55-4cd1-8341-c4d674fad0fa"). InnerVolumeSpecName "kube-api-access-vzwsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:46:30.392483 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.392459 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzwsk\" (UniqueName: \"kubernetes.io/projected/dff76af4-5e55-4cd1-8341-c4d674fad0fa-kube-api-access-vzwsk\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:30.864695 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.864663 4409 generic.go:356] "Generic (PLEG): container finished" podID="dff76af4-5e55-4cd1-8341-c4d674fad0fa" containerID="38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2" exitCode=0 Nov 03 07:46:30.864925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.864734 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" Nov 03 07:46:30.864925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.864750 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" event={"ID":"dff76af4-5e55-4cd1-8341-c4d674fad0fa","Type":"ContainerDied","Data":"38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2"} Nov 03 07:46:30.864925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.864814 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9" event={"ID":"dff76af4-5e55-4cd1-8341-c4d674fad0fa","Type":"ContainerDied","Data":"011777a73f119d95fc9afca9e58dab7c2e2903e663744b269bf4e4638d081b6e"} Nov 03 07:46:30.864925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.864845 4409 scope.go:122] "RemoveContainer" containerID="38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2" Nov 03 07:46:30.875069 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.875050 4409 scope.go:122] "RemoveContainer" containerID="38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2" Nov 03 07:46:30.875338 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:46:30.875320 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2\": container with ID starting with 38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2 not found: ID does not exist" containerID="38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2" Nov 03 07:46:30.875384 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.875345 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2"} err="failed to get container status \"38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2\": rpc error: code = NotFound desc = could not find container \"38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2\": container with ID starting with 38e5918a51c090db61e8c431aa7d80d01dfaf8e3438047d2e6eb25fafbf4cba2 not found: ID does not exist" Nov 03 07:46:30.911968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:30.911943 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547c2e42-d342-47b5-a9dd-d526027c7550" path="/var/lib/kubelet/pods/547c2e42-d342-47b5-a9dd-d526027c7550/volumes" Nov 03 07:46:31.120539 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:31.120470 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9"] Nov 03 07:46:31.127116 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:31.127090 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0e8bf4-7f5db668cf-z6ff9"] Nov 03 07:46:32.912261 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:32.912231 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff76af4-5e55-4cd1-8341-c4d674fad0fa" path="/var/lib/kubelet/pods/dff76af4-5e55-4cd1-8341-c4d674fad0fa/volumes" Nov 03 07:46:35.248831 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.248801 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg"] Nov 03 07:46:35.249295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249282 4409 cpu_manager.go:414] "containerMap: removing container" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" Nov 03 07:46:35.249344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249296 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" Nov 03 07:46:35.249344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249311 4409 cpu_manager.go:414] "containerMap: removing container" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" Nov 03 07:46:35.249344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249317 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" Nov 03 07:46:35.249344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249326 4409 cpu_manager.go:414] "containerMap: removing container" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerName="istio-proxy" Nov 03 07:46:35.249344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249331 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerName="istio-proxy" Nov 03 07:46:35.249344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249339 4409 cpu_manager.go:414] "containerMap: removing container" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" Nov 03 07:46:35.249344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249344 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249351 4409 cpu_manager.go:414] "containerMap: removing container" podUID="6a5d1906-d406-4178-8be5-2af12fc35c35" containerName="maas-api" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249356 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5d1906-d406-4178-8be5-2af12fc35c35" containerName="maas-api" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249364 4409 cpu_manager.go:414] "containerMap: removing container" podUID="c2810d11-55ad-45fb-8a49-4c5760fd60e8" containerName="payload-pre-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249369 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2810d11-55ad-45fb-8a49-4c5760fd60e8" containerName="payload-pre-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249380 4409 cpu_manager.go:414] "containerMap: removing container" podUID="547c2e42-d342-47b5-a9dd-d526027c7550" containerName="main" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249384 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547c2e42-d342-47b5-a9dd-d526027c7550" containerName="main" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249390 4409 cpu_manager.go:414] "containerMap: removing container" podUID="34efdb29-aee9-40f9-bb8e-92410e7b9e48" containerName="payload-pre-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249395 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34efdb29-aee9-40f9-bb8e-92410e7b9e48" containerName="payload-pre-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249402 4409 cpu_manager.go:414] "containerMap: removing container" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249405 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249413 4409 cpu_manager.go:414] "containerMap: removing container" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249417 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249425 4409 cpu_manager.go:414] "containerMap: removing container" podUID="43579285-f289-4bdc-8e96-94c3798f3de1" containerName="payload-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249430 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43579285-f289-4bdc-8e96-94c3798f3de1" containerName="payload-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249437 4409 cpu_manager.go:414] "containerMap: removing container" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249442 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249450 4409 cpu_manager.go:414] "containerMap: removing container" podUID="b1401ab1-a79a-43dc-9321-60679a32ca36" containerName="maas-api" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249454 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1401ab1-a79a-43dc-9321-60679a32ca36" containerName="maas-api" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249462 4409 cpu_manager.go:414] "containerMap: removing container" podUID="dff76af4-5e55-4cd1-8341-c4d674fad0fa" containerName="payload-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249466 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff76af4-5e55-4cd1-8341-c4d674fad0fa" containerName="payload-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249477 4409 cpu_manager.go:414] "containerMap: removing container" podUID="b01d19dc-7c01-4944-bf95-792f185c575c" containerName="payload-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249481 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01d19dc-7c01-4944-bf95-792f185c575c" containerName="payload-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249490 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9f03bf4c-6138-4f5a-a3c7-739cb83219bb" containerName="payload-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249494 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f03bf4c-6138-4f5a-a3c7-739cb83219bb" containerName="payload-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249502 4409 cpu_manager.go:414] "containerMap: removing container" podUID="b0432b16-3bc9-42dc-84b6-b685dd7d0a88" containerName="maas-api" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249508 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0432b16-3bc9-42dc-84b6-b685dd7d0a88" containerName="maas-api" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249516 4409 cpu_manager.go:414] "containerMap: removing container" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249520 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249526 4409 cpu_manager.go:414] "containerMap: removing container" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249530 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249537 4409 cpu_manager.go:414] "containerMap: removing container" podUID="93a5d08c-8abf-4e6f-8977-2485f249882c" containerName="payload-pre-processing" Nov 03 07:46:35.249572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249541 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a5d08c-8abf-4e6f-8977-2485f249882c" containerName="payload-pre-processing" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249605 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249613 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249618 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e54c4bd-d566-42ab-8613-67b296c03c18" containerName="istio-proxy" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249624 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c21443-7e7e-46c0-835b-ad5cb444ddcb" containerName="main" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249631 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5d1906-d406-4178-8be5-2af12fc35c35" containerName="maas-api" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249636 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2810d11-55ad-45fb-8a49-4c5760fd60e8" containerName="payload-pre-processing" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249642 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="547c2e42-d342-47b5-a9dd-d526027c7550" containerName="main" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249647 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="34efdb29-aee9-40f9-bb8e-92410e7b9e48" containerName="payload-pre-processing" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249652 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d7765c-f51f-43d5-9cbc-9cc1dd5dc36c" containerName="main" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249658 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="27976e39-e6a3-4269-966b-67cca0cf2d9e" containerName="main" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249663 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="43579285-f289-4bdc-8e96-94c3798f3de1" containerName="payload-processing" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249668 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249673 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1401ab1-a79a-43dc-9321-60679a32ca36" containerName="maas-api" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249678 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff76af4-5e55-4cd1-8341-c4d674fad0fa" containerName="payload-processing" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249683 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01d19dc-7c01-4944-bf95-792f185c575c" containerName="payload-processing" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249688 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f03bf4c-6138-4f5a-a3c7-739cb83219bb" containerName="payload-processing" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249694 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0432b16-3bc9-42dc-84b6-b685dd7d0a88" containerName="maas-api" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249699 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86da6dc-a180-4dd3-86e0-b34c28b47c37" containerName="istio-proxy" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249703 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0fec39-da43-4132-af60-9948d7f78cf3" containerName="main" Nov 03 07:46:35.250532 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.249722 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a5d08c-8abf-4e6f-8977-2485f249882c" containerName="payload-pre-processing" Nov 03 07:46:35.252917 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.252901 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.307641 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.307619 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1f2e7e-openshift-default-dockercfg-plh6k\"" Nov 03 07:46:35.345135 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.345110 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.345234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.345143 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.345234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.345171 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.345379 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.345360 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.345425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.345390 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.345459 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.345425 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1098b16-19e9-48a1-8a4a-507ab6b616de-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.345493 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.345461 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.345493 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.345487 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.345572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.345560 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd27g\" (UniqueName: \"kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-kube-api-access-fd27g\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.358644 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.358620 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg"] Nov 03 07:46:35.446365 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.446335 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd27g\" (UniqueName: \"kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-kube-api-access-fd27g\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.446495 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.446382 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.446495 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.446417 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.446495 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.446458 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.446747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.446595 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.446747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.446626 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.446747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.446668 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1098b16-19e9-48a1-8a4a-507ab6b616de-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.446747 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.446700 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.446977 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.446755 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.446977 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.446847 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-data\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.446977 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.446895 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-credential-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.447160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.447135 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-certs\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.447218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.447193 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-socket\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.447472 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.447451 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1098b16-19e9-48a1-8a4a-507ab6b616de-istiod-ca-cert\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.449061 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.449039 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-envoy\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.449244 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.449226 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-podinfo\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.509469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.509414 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-token\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.509585 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.509563 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd27g\" (UniqueName: \"kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-kube-api-access-fd27g\") pod \"e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.566655 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.566629 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:35.617300 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:35.617248 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" podUID="34261966-3f63-47b2-9bff-e687beb22e3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 07:46:36.006591 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:36.006568 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg"] Nov 03 07:46:36.007798 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:46:36.007776 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1098b16_19e9_48a1_8a4a_507ab6b616de.slice/crio-67c8ff3f9c7d78a949e840f96d31547a55a38b35434df8a8e9e5e64c66c16df6 WatchSource:0}: Error finding container 67c8ff3f9c7d78a949e840f96d31547a55a38b35434df8a8e9e5e64c66c16df6: Status 404 returned error can't find the container with id 67c8ff3f9c7d78a949e840f96d31547a55a38b35434df8a8e9e5e64c66c16df6 Nov 03 07:46:36.010016 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:36.009986 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:46:36.010092 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:36.010045 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:46:36.010092 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:36.010071 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:46:36.901672 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:36.901624 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" event={"ID":"d1098b16-19e9-48a1-8a4a-507ab6b616de","Type":"ContainerStarted","Data":"7b816afa56023d706b36f2c43c09420a1bc64b59b4198a28abd24a75e129355f"} Nov 03 07:46:36.902205 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:36.901761 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" event={"ID":"d1098b16-19e9-48a1-8a4a-507ab6b616de","Type":"ContainerStarted","Data":"67c8ff3f9c7d78a949e840f96d31547a55a38b35434df8a8e9e5e64c66c16df6"} Nov 03 07:46:37.027707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:37.027649 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" podStartSLOduration=2.027635407 podStartE2EDuration="2.027635407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:46:37.026045565 +0000 UTC m=+2852.218587328" watchObservedRunningTime="2026-11-03 07:46:37.027635407 +0000 UTC m=+2852.220177156" Nov 03 07:46:37.577738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:37.576154 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:37.588738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:37.580830 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 07:46:37.588738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:37.580895 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" podUID="d1098b16-19e9-48a1-8a4a-507ab6b616de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 07:46:38.035588 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.035499 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk"] Nov 03 07:46:38.036113 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.035696 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" podUID="24e61dd8-0436-41bd-b80e-bdfd1f70b16b" containerName="maas-api" containerID="cri-o://e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25" gracePeriod=30 Nov 03 07:46:38.400215 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.400186 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:46:38.434134 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.432945 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-kube-api-access-pxn5z\" (UniqueName: \"kubernetes.io/projected/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-kube-api-access-pxn5z\") pod \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " Nov 03 07:46:38.434134 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.433095 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-metrics-certs\") pod \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " Nov 03 07:46:38.434134 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.433150 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-openshift-service-ca\") pod \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " Nov 03 07:46:38.434134 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.433227 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-maas-api-tls\") pod \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\" (UID: \"24e61dd8-0436-41bd-b80e-bdfd1f70b16b\") " Nov 03 07:46:38.435107 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.435079 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-openshift-service-ca" pod "24e61dd8-0436-41bd-b80e-bdfd1f70b16b" (UID: "24e61dd8-0436-41bd-b80e-bdfd1f70b16b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:46:38.440109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.440060 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-metrics-certs" pod "24e61dd8-0436-41bd-b80e-bdfd1f70b16b" (UID: "24e61dd8-0436-41bd-b80e-bdfd1f70b16b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:46:38.440803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.440764 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-maas-api-tls" pod "24e61dd8-0436-41bd-b80e-bdfd1f70b16b" (UID: "24e61dd8-0436-41bd-b80e-bdfd1f70b16b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:46:38.485739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.477731 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-kube-api-access-pxn5z" pod "24e61dd8-0436-41bd-b80e-bdfd1f70b16b" (UID: "24e61dd8-0436-41bd-b80e-bdfd1f70b16b"). InnerVolumeSpecName "kube-api-access-pxn5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:46:38.534382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.534343 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxn5z\" (UniqueName: \"kubernetes.io/projected/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-kube-api-access-pxn5z\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:38.534382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.534380 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:38.534382 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.534391 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:38.534619 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.534405 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24e61dd8-0436-41bd-b80e-bdfd1f70b16b-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:38.590742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.589140 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 07:46:38.590742 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.589215 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" podUID="d1098b16-19e9-48a1-8a4a-507ab6b616de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 07:46:38.798724 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:46:38.798629 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:46:38.919017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.918969 4409 generic.go:356] "Generic (PLEG): container finished" podID="24e61dd8-0436-41bd-b80e-bdfd1f70b16b" containerID="e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25" exitCode=0 Nov 03 07:46:38.919017 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.919019 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" Nov 03 07:46:38.919230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.919074 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" event={"ID":"24e61dd8-0436-41bd-b80e-bdfd1f70b16b","Type":"ContainerDied","Data":"e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25"} Nov 03 07:46:38.919230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.919124 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk" event={"ID":"24e61dd8-0436-41bd-b80e-bdfd1f70b16b","Type":"ContainerDied","Data":"437095b04ab566c7ccdfc5c7f388b9ce85c444e058620829a05c72f636957502"} Nov 03 07:46:38.919230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.919150 4409 scope.go:122] "RemoveContainer" containerID="e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25" Nov 03 07:46:38.930803 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.930774 4409 scope.go:122] "RemoveContainer" containerID="e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25" Nov 03 07:46:38.931170 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:46:38.931089 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25\": container with ID starting with e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25 not found: ID does not exist" containerID="e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25" Nov 03 07:46:38.931170 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:38.931146 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25"} err="failed to get container status \"e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25\": rpc error: code = NotFound desc = could not find container \"e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25\": container with ID starting with e49adc4b12c14b35c1d8b6ddccdc4a60d3828a10b7edb0a5b7d237a5bd3f6c25 not found: ID does not exist" Nov 03 07:46:39.113690 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:39.113627 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk"] Nov 03 07:46:39.115001 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:39.114969 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0e8bf4-886d78d59-lzphk"] Nov 03 07:46:39.590740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:39.589135 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 07:46:39.590740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:39.589240 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" podUID="d1098b16-19e9-48a1-8a4a-507ab6b616de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 07:46:40.577740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:40.572148 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 07:46:40.577740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:40.572210 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" podUID="d1098b16-19e9-48a1-8a4a-507ab6b616de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 07:46:40.913641 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:40.913553 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e61dd8-0436-41bd-b80e-bdfd1f70b16b" path="/var/lib/kubelet/pods/24e61dd8-0436-41bd-b80e-bdfd1f70b16b/volumes" Nov 03 07:46:41.595891 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:41.595854 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 07:46:41.596358 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:41.595936 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" podUID="d1098b16-19e9-48a1-8a4a-507ab6b616de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 07:46:42.568014 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:42.567973 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 07:46:42.568194 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:42.568059 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" podUID="d1098b16-19e9-48a1-8a4a-507ab6b616de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 07:46:44.585289 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:44.585250 4409 patch_prober.go:29] interesting pod/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:46:44.585785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:44.585344 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" podUID="d1098b16-19e9-48a1-8a4a-507ab6b616de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:46:44.773190 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:44.773153 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:44.773793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:44.773770 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:44.774779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:44.774696 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:45.630174 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:45.630132 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" podUID="34261966-3f63-47b2-9bff-e687beb22e3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 07:46:47.812777 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:47.812737 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg"] Nov 03 07:46:47.813202 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:47.812968 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" podUID="d1098b16-19e9-48a1-8a4a-507ab6b616de" containerName="istio-proxy" containerID="cri-o://7b816afa56023d706b36f2c43c09420a1bc64b59b4198a28abd24a75e129355f" gracePeriod=30 Nov 03 07:46:49.439746 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:49.439674 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s"] Nov 03 07:46:49.440189 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:49.439914 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" containerID="cri-o://e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b" gracePeriod=30 Nov 03 07:46:50.259191 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:50.259135 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" Nov 03 07:46:50.331279 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:50.331249 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34261966-3f63-47b2-9bff-e687beb22e3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34261966-3f63-47b2-9bff-e687beb22e3e-plugins-config-volume\") pod \"34261966-3f63-47b2-9bff-e687beb22e3e\" (UID: \"34261966-3f63-47b2-9bff-e687beb22e3e\") " Nov 03 07:46:50.331433 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:50.331316 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34261966-3f63-47b2-9bff-e687beb22e3e-kube-api-access-xg47p\" (UniqueName: \"kubernetes.io/projected/34261966-3f63-47b2-9bff-e687beb22e3e-kube-api-access-xg47p\") pod \"34261966-3f63-47b2-9bff-e687beb22e3e\" (UID: \"34261966-3f63-47b2-9bff-e687beb22e3e\") " Nov 03 07:46:50.331809 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:50.331787 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34261966-3f63-47b2-9bff-e687beb22e3e-plugins-config-volume" pod "34261966-3f63-47b2-9bff-e687beb22e3e" (UID: "34261966-3f63-47b2-9bff-e687beb22e3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:46:50.331907 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:50.331885 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34261966-3f63-47b2-9bff-e687beb22e3e-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:50.333836 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:50.333797 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34261966-3f63-47b2-9bff-e687beb22e3e-kube-api-access-xg47p" pod "34261966-3f63-47b2-9bff-e687beb22e3e" (UID: "34261966-3f63-47b2-9bff-e687beb22e3e"). InnerVolumeSpecName "kube-api-access-xg47p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:46:50.432639 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:50.432605 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg47p\" (UniqueName: \"kubernetes.io/projected/34261966-3f63-47b2-9bff-e687beb22e3e-kube-api-access-xg47p\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:50.608779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:50.608741 4409 patch_prober.go:29] interesting pod/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:46:50.609184 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:50.608791 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:46:51.013051 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:51.013019 4409 generic.go:356] "Generic (PLEG): container finished" podID="34261966-3f63-47b2-9bff-e687beb22e3e" containerID="3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02" exitCode=0 Nov 03 07:46:51.013247 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:51.013076 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" Nov 03 07:46:51.013247 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:51.013080 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" event={"ID":"34261966-3f63-47b2-9bff-e687beb22e3e","Type":"ContainerDied","Data":"3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02"} Nov 03 07:46:51.013247 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:51.013136 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7" event={"ID":"34261966-3f63-47b2-9bff-e687beb22e3e","Type":"ContainerDied","Data":"ddd188d62813bfa15423dc8fdb09d66a2e0e2c0ef67f8ac746c0bb71ba6be4dd"} Nov 03 07:46:51.013247 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:51.013161 4409 scope.go:122] "RemoveContainer" containerID="3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02" Nov 03 07:46:51.022652 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:51.022633 4409 scope.go:122] "RemoveContainer" containerID="3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02" Nov 03 07:46:51.022957 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:46:51.022935 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02\": container with ID starting with 3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02 not found: ID does not exist" containerID="3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02" Nov 03 07:46:51.023012 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:51.022965 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02"} err="failed to get container status \"3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02\": rpc error: code = NotFound desc = could not find container \"3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02\": container with ID starting with 3cc7ed0e19fcd325e1f0e9ba7314c2a95f2734e97839a9f19fe3aa122f4f7d02 not found: ID does not exist" Nov 03 07:46:51.181992 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:51.181915 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7"] Nov 03 07:46:51.231809 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:51.231778 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0e8bf4-7ddf9dfcdc-27jz7"] Nov 03 07:46:52.912544 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:52.912519 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34261966-3f63-47b2-9bff-e687beb22e3e" path="/var/lib/kubelet/pods/34261966-3f63-47b2-9bff-e687beb22e3e/volumes" Nov 03 07:46:53.026258 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.026224 4409 generic.go:356] "Generic (PLEG): container finished" podID="d1098b16-19e9-48a1-8a4a-507ab6b616de" containerID="7b816afa56023d706b36f2c43c09420a1bc64b59b4198a28abd24a75e129355f" exitCode=0 Nov 03 07:46:53.026447 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.026409 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" event={"ID":"d1098b16-19e9-48a1-8a4a-507ab6b616de","Type":"ContainerDied","Data":"7b816afa56023d706b36f2c43c09420a1bc64b59b4198a28abd24a75e129355f"} Nov 03 07:46:53.079608 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.079578 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:53.162966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.162907 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-token\" (UniqueName: \"kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-token\") pod \"d1098b16-19e9-48a1-8a4a-507ab6b616de\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " Nov 03 07:46:53.163067 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163000 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-kube-api-access-fd27g\" (UniqueName: \"kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-kube-api-access-fd27g\") pod \"d1098b16-19e9-48a1-8a4a-507ab6b616de\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " Nov 03 07:46:53.163067 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163023 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-envoy\") pod \"d1098b16-19e9-48a1-8a4a-507ab6b616de\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " Nov 03 07:46:53.163067 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163043 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-podinfo\") pod \"d1098b16-19e9-48a1-8a4a-507ab6b616de\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " Nov 03 07:46:53.163230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163072 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-credential-socket\") pod \"d1098b16-19e9-48a1-8a4a-507ab6b616de\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " Nov 03 07:46:53.163230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163144 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1098b16-19e9-48a1-8a4a-507ab6b616de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1098b16-19e9-48a1-8a4a-507ab6b616de-istiod-ca-cert\") pod \"d1098b16-19e9-48a1-8a4a-507ab6b616de\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " Nov 03 07:46:53.163230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163190 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-certs\") pod \"d1098b16-19e9-48a1-8a4a-507ab6b616de\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " Nov 03 07:46:53.163389 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163261 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-data\") pod \"d1098b16-19e9-48a1-8a4a-507ab6b616de\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " Nov 03 07:46:53.163389 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163313 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-socket\") pod \"d1098b16-19e9-48a1-8a4a-507ab6b616de\" (UID: \"d1098b16-19e9-48a1-8a4a-507ab6b616de\") " Nov 03 07:46:53.164072 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163538 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-credential-socket" pod "d1098b16-19e9-48a1-8a4a-507ab6b616de" (UID: "d1098b16-19e9-48a1-8a4a-507ab6b616de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:53.164072 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163607 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1098b16-19e9-48a1-8a4a-507ab6b616de-istiod-ca-cert" pod "d1098b16-19e9-48a1-8a4a-507ab6b616de" (UID: "d1098b16-19e9-48a1-8a4a-507ab6b616de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:46:53.164072 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163971 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-certs" pod "d1098b16-19e9-48a1-8a4a-507ab6b616de" (UID: "d1098b16-19e9-48a1-8a4a-507ab6b616de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:53.164072 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163992 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:53.164072 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.164017 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1098b16-19e9-48a1-8a4a-507ab6b616de-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:53.164072 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.163991 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-socket" pod "d1098b16-19e9-48a1-8a4a-507ab6b616de" (UID: "d1098b16-19e9-48a1-8a4a-507ab6b616de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:53.164496 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.164094 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-data" pod "d1098b16-19e9-48a1-8a4a-507ab6b616de" (UID: "d1098b16-19e9-48a1-8a4a-507ab6b616de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:53.165326 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.165305 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-token" pod "d1098b16-19e9-48a1-8a4a-507ab6b616de" (UID: "d1098b16-19e9-48a1-8a4a-507ab6b616de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:46:53.165443 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.165345 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-envoy" pod "d1098b16-19e9-48a1-8a4a-507ab6b616de" (UID: "d1098b16-19e9-48a1-8a4a-507ab6b616de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:53.165443 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.165405 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-podinfo" pod "d1098b16-19e9-48a1-8a4a-507ab6b616de" (UID: "d1098b16-19e9-48a1-8a4a-507ab6b616de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:46:53.165519 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.165490 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-kube-api-access-fd27g" pod "d1098b16-19e9-48a1-8a4a-507ab6b616de" (UID: "d1098b16-19e9-48a1-8a4a-507ab6b616de"). InnerVolumeSpecName "kube-api-access-fd27g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:46:53.265420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.265392 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:53.265420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.265418 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:53.265656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.265433 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd27g\" (UniqueName: \"kubernetes.io/projected/d1098b16-19e9-48a1-8a4a-507ab6b616de-kube-api-access-fd27g\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:53.265656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.265443 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:53.265656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.265452 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:53.265656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.265460 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:53.265656 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:53.265469 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1098b16-19e9-48a1-8a4a-507ab6b616de-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:54.033900 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.033870 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" Nov 03 07:46:54.034314 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.033900 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg" event={"ID":"d1098b16-19e9-48a1-8a4a-507ab6b616de","Type":"ContainerDied","Data":"67c8ff3f9c7d78a949e840f96d31547a55a38b35434df8a8e9e5e64c66c16df6"} Nov 03 07:46:54.034314 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.033968 4409 scope.go:122] "RemoveContainer" containerID="7b816afa56023d706b36f2c43c09420a1bc64b59b4198a28abd24a75e129355f" Nov 03 07:46:54.251612 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.251584 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg"] Nov 03 07:46:54.301178 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.301096 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1f2e7e-openshift-default-7c465d49bc-sm9rg"] Nov 03 07:46:54.683596 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.683569 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:46:54.779680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.779651 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-envoy\") pod \"d816bfac-359f-463e-81b9-accd32ce79e1\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " Nov 03 07:46:54.779849 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.779688 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-data\") pod \"d816bfac-359f-463e-81b9-accd32ce79e1\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " Nov 03 07:46:54.779849 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.779741 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-kube-api-access-pg6c4\" (UniqueName: \"kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-kube-api-access-pg6c4\") pod \"d816bfac-359f-463e-81b9-accd32ce79e1\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " Nov 03 07:46:54.779849 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.779799 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-credential-socket\") pod \"d816bfac-359f-463e-81b9-accd32ce79e1\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " Nov 03 07:46:54.780047 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.779873 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-istio-token\" (UniqueName: \"kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-istio-token\") pod \"d816bfac-359f-463e-81b9-accd32ce79e1\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " Nov 03 07:46:54.780047 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.779902 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-socket\") pod \"d816bfac-359f-463e-81b9-accd32ce79e1\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " Nov 03 07:46:54.780047 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.779939 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d816bfac-359f-463e-81b9-accd32ce79e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d816bfac-359f-463e-81b9-accd32ce79e1-istiod-ca-cert\") pod \"d816bfac-359f-463e-81b9-accd32ce79e1\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " Nov 03 07:46:54.780047 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.780005 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-certs\") pod \"d816bfac-359f-463e-81b9-accd32ce79e1\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " Nov 03 07:46:54.780047 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.780006 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-data" pod "d816bfac-359f-463e-81b9-accd32ce79e1" (UID: "d816bfac-359f-463e-81b9-accd32ce79e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:54.780047 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.780042 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d816bfac-359f-463e-81b9-accd32ce79e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d816bfac-359f-463e-81b9-accd32ce79e1-istio-podinfo\") pod \"d816bfac-359f-463e-81b9-accd32ce79e1\" (UID: \"d816bfac-359f-463e-81b9-accd32ce79e1\") " Nov 03 07:46:54.780551 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.780173 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-socket" pod "d816bfac-359f-463e-81b9-accd32ce79e1" (UID: "d816bfac-359f-463e-81b9-accd32ce79e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:54.780551 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.780181 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-credential-socket" pod "d816bfac-359f-463e-81b9-accd32ce79e1" (UID: "d816bfac-359f-463e-81b9-accd32ce79e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:54.780551 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.780311 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-certs" pod "d816bfac-359f-463e-81b9-accd32ce79e1" (UID: "d816bfac-359f-463e-81b9-accd32ce79e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:54.780551 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.780352 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d816bfac-359f-463e-81b9-accd32ce79e1-istiod-ca-cert" pod "d816bfac-359f-463e-81b9-accd32ce79e1" (UID: "d816bfac-359f-463e-81b9-accd32ce79e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:46:54.780551 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.780527 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:54.780551 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.780540 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d816bfac-359f-463e-81b9-accd32ce79e1-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:54.780551 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.780549 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:54.780551 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.780557 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:54.781028 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.780566 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:54.782127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.782102 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-envoy" pod "d816bfac-359f-463e-81b9-accd32ce79e1" (UID: "d816bfac-359f-463e-81b9-accd32ce79e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:46:54.782127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.782106 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-kube-api-access-pg6c4" pod "d816bfac-359f-463e-81b9-accd32ce79e1" (UID: "d816bfac-359f-463e-81b9-accd32ce79e1"). InnerVolumeSpecName "kube-api-access-pg6c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:46:54.782280 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.782202 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d816bfac-359f-463e-81b9-accd32ce79e1-istio-podinfo" pod "d816bfac-359f-463e-81b9-accd32ce79e1" (UID: "d816bfac-359f-463e-81b9-accd32ce79e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:46:54.782450 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.782432 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-istio-token" pod "d816bfac-359f-463e-81b9-accd32ce79e1" (UID: "d816bfac-359f-463e-81b9-accd32ce79e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:46:54.881779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.881684 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d816bfac-359f-463e-81b9-accd32ce79e1-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:54.881779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.881739 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d816bfac-359f-463e-81b9-accd32ce79e1-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:54.881779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.881750 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg6c4\" (UniqueName: \"kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-kube-api-access-pg6c4\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:54.881779 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.881761 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d816bfac-359f-463e-81b9-accd32ce79e1-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:46:54.912827 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:54.912796 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1098b16-19e9-48a1-8a4a-507ab6b616de" path="/var/lib/kubelet/pods/d1098b16-19e9-48a1-8a4a-507ab6b616de/volumes" Nov 03 07:46:55.041396 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:55.041362 4409 generic.go:356] "Generic (PLEG): container finished" podID="d816bfac-359f-463e-81b9-accd32ce79e1" containerID="e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b" exitCode=0 Nov 03 07:46:55.041396 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:55.041399 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" event={"ID":"d816bfac-359f-463e-81b9-accd32ce79e1","Type":"ContainerDied","Data":"e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b"} Nov 03 07:46:55.041883 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:55.041409 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" Nov 03 07:46:55.041883 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:55.041440 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s" event={"ID":"d816bfac-359f-463e-81b9-accd32ce79e1","Type":"ContainerDied","Data":"f2b97ec0c481f956e3899a673ded9983fc7b1b864a502e9c6b2489c19e9c508f"} Nov 03 07:46:55.041883 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:55.041462 4409 scope.go:122] "RemoveContainer" containerID="e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b" Nov 03 07:46:55.051185 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:55.051160 4409 scope.go:122] "RemoveContainer" containerID="bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11" Nov 03 07:46:55.059855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:55.059832 4409 scope.go:122] "RemoveContainer" containerID="e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b" Nov 03 07:46:55.060153 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:46:55.060134 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b\": container with ID starting with e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b not found: ID does not exist" containerID="e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b" Nov 03 07:46:55.060231 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:55.060159 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b"} err="failed to get container status \"e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b\": rpc error: code = NotFound desc = could not find container \"e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b\": container with ID starting with e75fbb3e2d9673521165ba29a15573411025ca764065b0df241f7e7f3397b08b not found: ID does not exist" Nov 03 07:46:55.060231 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:55.060176 4409 scope.go:122] "RemoveContainer" containerID="bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11" Nov 03 07:46:55.060492 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:46:55.060469 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11\": container with ID starting with bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11 not found: ID does not exist" containerID="bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11" Nov 03 07:46:55.060537 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:55.060498 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11"} err="failed to get container status \"bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11\": rpc error: code = NotFound desc = could not find container \"bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11\": container with ID starting with bc5bc17249a127298f664096b14866e0950fa8ca384b92b168dbef3555386a11 not found: ID does not exist" Nov 03 07:46:55.210331 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:55.210259 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s"] Nov 03 07:46:55.259863 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:55.259835 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0e8bf4-openshift-default-575655668f-2w77s"] Nov 03 07:46:56.915385 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:46:56.915354 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" path="/var/lib/kubelet/pods/d816bfac-359f-463e-81b9-accd32ce79e1/volumes" Nov 03 07:47:02.003134 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:02.003102 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r"] Nov 03 07:47:05.791357 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:05.791317 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh"] Nov 03 07:47:05.791861 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:05.791576 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" podUID="79c28244-4715-46d9-9162-aeaa1d92dc1e" containerName="maas-api" containerID="cri-o://b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035" gracePeriod=30 Nov 03 07:47:06.072706 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.072680 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:47:06.111834 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.111803 4409 generic.go:356] "Generic (PLEG): container finished" podID="79c28244-4715-46d9-9162-aeaa1d92dc1e" containerID="b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035" exitCode=0 Nov 03 07:47:06.111959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.111855 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" Nov 03 07:47:06.111959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.111891 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" event={"ID":"79c28244-4715-46d9-9162-aeaa1d92dc1e","Type":"ContainerDied","Data":"b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035"} Nov 03 07:47:06.111959 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.111948 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh" event={"ID":"79c28244-4715-46d9-9162-aeaa1d92dc1e","Type":"ContainerDied","Data":"7710bc375bcbee503379ad94cd3feecbc5c35e9e2d51ad8ac875660c51e91a50"} Nov 03 07:47:06.112065 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.111972 4409 scope.go:122] "RemoveContainer" containerID="b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035" Nov 03 07:47:06.120420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.120395 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-maas-api-tls\") pod \"79c28244-4715-46d9-9162-aeaa1d92dc1e\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " Nov 03 07:47:06.120536 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.120438 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-metrics-certs\") pod \"79c28244-4715-46d9-9162-aeaa1d92dc1e\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " Nov 03 07:47:06.120536 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.120486 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c28244-4715-46d9-9162-aeaa1d92dc1e-kube-api-access-pwxs7\" (UniqueName: \"kubernetes.io/projected/79c28244-4715-46d9-9162-aeaa1d92dc1e-kube-api-access-pwxs7\") pod \"79c28244-4715-46d9-9162-aeaa1d92dc1e\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " Nov 03 07:47:06.120740 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.120693 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c28244-4715-46d9-9162-aeaa1d92dc1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c28244-4715-46d9-9162-aeaa1d92dc1e-openshift-service-ca\") pod \"79c28244-4715-46d9-9162-aeaa1d92dc1e\" (UID: \"79c28244-4715-46d9-9162-aeaa1d92dc1e\") " Nov 03 07:47:06.121372 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.121340 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c28244-4715-46d9-9162-aeaa1d92dc1e-openshift-service-ca" pod "79c28244-4715-46d9-9162-aeaa1d92dc1e" (UID: "79c28244-4715-46d9-9162-aeaa1d92dc1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:47:06.121910 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.121888 4409 scope.go:122] "RemoveContainer" containerID="b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035" Nov 03 07:47:06.122269 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:47:06.122236 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035\": container with ID starting with b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035 not found: ID does not exist" containerID="b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035" Nov 03 07:47:06.122375 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.122280 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035"} err="failed to get container status \"b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035\": rpc error: code = NotFound desc = could not find container \"b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035\": container with ID starting with b4c9bc59629895fcb49612b11942a1229232550e9d21e56d5465f6f10f45f035 not found: ID does not exist" Nov 03 07:47:06.122644 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.122620 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-maas-api-tls" pod "79c28244-4715-46d9-9162-aeaa1d92dc1e" (UID: "79c28244-4715-46d9-9162-aeaa1d92dc1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:47:06.122826 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.122804 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c28244-4715-46d9-9162-aeaa1d92dc1e-kube-api-access-pwxs7" pod "79c28244-4715-46d9-9162-aeaa1d92dc1e" (UID: "79c28244-4715-46d9-9162-aeaa1d92dc1e"). InnerVolumeSpecName "kube-api-access-pwxs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:06.122923 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.122867 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-metrics-certs" pod "79c28244-4715-46d9-9162-aeaa1d92dc1e" (UID: "79c28244-4715-46d9-9162-aeaa1d92dc1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:47:06.222182 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.222150 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:06.222182 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.222176 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79c28244-4715-46d9-9162-aeaa1d92dc1e-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:06.222182 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.222186 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwxs7\" (UniqueName: \"kubernetes.io/projected/79c28244-4715-46d9-9162-aeaa1d92dc1e-kube-api-access-pwxs7\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:06.222499 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.222196 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79c28244-4715-46d9-9162-aeaa1d92dc1e-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:06.536754 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.536722 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh"] Nov 03 07:47:06.542232 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.542209 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-669d6b-64d785fff4-xp8dh"] Nov 03 07:47:06.913136 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:06.913105 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c28244-4715-46d9-9162-aeaa1d92dc1e" path="/var/lib/kubelet/pods/79c28244-4715-46d9-9162-aeaa1d92dc1e/volumes" Nov 03 07:47:07.004375 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:07.004331 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" podUID="53c50c34-72c7-4e1b-a926-4e9ef47b9fd5" containerName="payload-pre-processing" containerID="cri-o://050790d2157c24854bc0d57c2e21ba969f3f8df015ddd4296a99874beaec7ea7" gracePeriod=25 Nov 03 07:47:08.773839 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:47:08.773796 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:47:12.268601 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:12.268561 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" podUID="53c50c34-72c7-4e1b-a926-4e9ef47b9fd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 07:47:22.268685 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:22.268642 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" podUID="53c50c34-72c7-4e1b-a926-4e9ef47b9fd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 07:47:27.223980 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:27.223946 4409 generic.go:356] "Generic (PLEG): container finished" podID="53c50c34-72c7-4e1b-a926-4e9ef47b9fd5" containerID="050790d2157c24854bc0d57c2e21ba969f3f8df015ddd4296a99874beaec7ea7" exitCode=0 Nov 03 07:47:27.224337 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:27.224063 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" event={"ID":"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5","Type":"ContainerDied","Data":"050790d2157c24854bc0d57c2e21ba969f3f8df015ddd4296a99874beaec7ea7"} Nov 03 07:47:27.277417 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:27.277349 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" Nov 03 07:47:27.357793 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:27.357756 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-kube-api-access-lgrhn\" (UniqueName: \"kubernetes.io/projected/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-kube-api-access-lgrhn\") pod \"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5\" (UID: \"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5\") " Nov 03 07:47:27.357989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:27.357834 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-plugins-config-volume\") pod \"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5\" (UID: \"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5\") " Nov 03 07:47:27.358208 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:27.358184 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-plugins-config-volume" pod "53c50c34-72c7-4e1b-a926-4e9ef47b9fd5" (UID: "53c50c34-72c7-4e1b-a926-4e9ef47b9fd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:47:27.360181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:27.360151 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-kube-api-access-lgrhn" pod "53c50c34-72c7-4e1b-a926-4e9ef47b9fd5" (UID: "53c50c34-72c7-4e1b-a926-4e9ef47b9fd5"). InnerVolumeSpecName "kube-api-access-lgrhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:27.458668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:27.458636 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:27.458668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:27.458665 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgrhn\" (UniqueName: \"kubernetes.io/projected/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5-kube-api-access-lgrhn\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:28.230263 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:28.230232 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" Nov 03 07:47:28.230263 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:28.230249 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r" event={"ID":"53c50c34-72c7-4e1b-a926-4e9ef47b9fd5","Type":"ContainerDied","Data":"cd0da443f9bf029b3cc5bc2ff57bd23cb2e7529cdc31f6b83f13876bc343f866"} Nov 03 07:47:28.230924 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:28.230317 4409 scope.go:122] "RemoveContainer" containerID="050790d2157c24854bc0d57c2e21ba969f3f8df015ddd4296a99874beaec7ea7" Nov 03 07:47:28.398053 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:28.398014 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r"] Nov 03 07:47:28.406416 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:28.406383 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dcdb23-6b4cc475b7-qf28r"] Nov 03 07:47:28.912451 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:28.912424 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c50c34-72c7-4e1b-a926-4e9ef47b9fd5" path="/var/lib/kubelet/pods/53c50c34-72c7-4e1b-a926-4e9ef47b9fd5/volumes" Nov 03 07:47:38.774655 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:47:38.774615 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:47:43.247607 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:43.247558 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6"] Nov 03 07:47:43.248104 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:43.247844 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" containerID="cri-o://5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c" gracePeriod=30 Nov 03 07:47:43.249637 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:43.249613 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g"] Nov 03 07:47:43.249855 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:43.249835 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" containerID="cri-o://308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82" gracePeriod=30 Nov 03 07:47:43.383667 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:43.383627 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gp9f6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:47:43.383864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:43.383696 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:47:43.788302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:43.788240 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-s4f8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:47:43.788526 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:43.788315 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:47:48.506813 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.506785 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:47:48.537064 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.537039 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:47:48.627486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627463 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-envoy\") pod \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " Nov 03 07:47:48.627587 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627499 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/620abc13-9e8b-440f-a23a-f230d0656275-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620abc13-9e8b-440f-a23a-f230d0656275-istio-podinfo\") pod \"620abc13-9e8b-440f-a23a-f230d0656275\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " Nov 03 07:47:48.627587 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627521 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-certs\") pod \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " Nov 03 07:47:48.627587 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627550 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-data\") pod \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " Nov 03 07:47:48.627783 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627586 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-kube-api-access-nx9nz\" (UniqueName: \"kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-kube-api-access-nx9nz\") pod \"620abc13-9e8b-440f-a23a-f230d0656275\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " Nov 03 07:47:48.627783 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627617 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-token\" (UniqueName: \"kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-token\") pod \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " Nov 03 07:47:48.627783 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627661 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620abc13-9e8b-440f-a23a-f230d0656275-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620abc13-9e8b-440f-a23a-f230d0656275-istiod-ca-cert\") pod \"620abc13-9e8b-440f-a23a-f230d0656275\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " Nov 03 07:47:48.627783 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627696 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-kube-api-access-jzqmh\" (UniqueName: \"kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-kube-api-access-jzqmh\") pod \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " Nov 03 07:47:48.627783 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627750 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-data\") pod \"620abc13-9e8b-440f-a23a-f230d0656275\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " Nov 03 07:47:48.628063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627790 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-certs\") pod \"620abc13-9e8b-440f-a23a-f230d0656275\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " Nov 03 07:47:48.628063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627801 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-certs" pod "d54db83b-2f0a-46d1-90de-84c048ff4e0d" (UID: "d54db83b-2f0a-46d1-90de-84c048ff4e0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.628063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627820 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-socket\") pod \"620abc13-9e8b-440f-a23a-f230d0656275\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " Nov 03 07:47:48.628063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627895 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-socket\") pod \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " Nov 03 07:47:48.628063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627942 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-envoy\") pod \"620abc13-9e8b-440f-a23a-f230d0656275\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " Nov 03 07:47:48.628063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627981 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-podinfo\") pod \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " Nov 03 07:47:48.628063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.628039 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-istio-token\" (UniqueName: \"kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-istio-token\") pod \"620abc13-9e8b-440f-a23a-f230d0656275\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " Nov 03 07:47:48.628444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.628087 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-credential-socket\") pod \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " Nov 03 07:47:48.628444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.628154 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-credential-socket\") pod \"620abc13-9e8b-440f-a23a-f230d0656275\" (UID: \"620abc13-9e8b-440f-a23a-f230d0656275\") " Nov 03 07:47:48.628444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.628349 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istiod-ca-cert\") pod \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\" (UID: \"d54db83b-2f0a-46d1-90de-84c048ff4e0d\") " Nov 03 07:47:48.628972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.627889 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-data" pod "d54db83b-2f0a-46d1-90de-84c048ff4e0d" (UID: "d54db83b-2f0a-46d1-90de-84c048ff4e0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.628972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.628409 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620abc13-9e8b-440f-a23a-f230d0656275-istiod-ca-cert" pod "620abc13-9e8b-440f-a23a-f230d0656275" (UID: "620abc13-9e8b-440f-a23a-f230d0656275"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:47:48.628972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.628539 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-certs" pod "620abc13-9e8b-440f-a23a-f230d0656275" (UID: "620abc13-9e8b-440f-a23a-f230d0656275"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.628972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.628561 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-data" pod "620abc13-9e8b-440f-a23a-f230d0656275" (UID: "620abc13-9e8b-440f-a23a-f230d0656275"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.628972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.628795 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-socket" pod "d54db83b-2f0a-46d1-90de-84c048ff4e0d" (UID: "d54db83b-2f0a-46d1-90de-84c048ff4e0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.629492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.628993 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-credential-socket" pod "d54db83b-2f0a-46d1-90de-84c048ff4e0d" (UID: "d54db83b-2f0a-46d1-90de-84c048ff4e0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.629492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.629007 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-credential-socket" pod "620abc13-9e8b-440f-a23a-f230d0656275" (UID: "620abc13-9e8b-440f-a23a-f230d0656275"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.629492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.629049 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.629492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.629165 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istiod-ca-cert" pod "d54db83b-2f0a-46d1-90de-84c048ff4e0d" (UID: "d54db83b-2f0a-46d1-90de-84c048ff4e0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:47:48.629853 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.629826 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-socket" pod "620abc13-9e8b-440f-a23a-f230d0656275" (UID: "620abc13-9e8b-440f-a23a-f230d0656275"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.630439 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.630413 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/620abc13-9e8b-440f-a23a-f230d0656275-istio-podinfo" pod "620abc13-9e8b-440f-a23a-f230d0656275" (UID: "620abc13-9e8b-440f-a23a-f230d0656275"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:47:48.630683 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.630652 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-envoy" pod "620abc13-9e8b-440f-a23a-f230d0656275" (UID: "620abc13-9e8b-440f-a23a-f230d0656275"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.631113 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.631080 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-kube-api-access-nx9nz" pod "620abc13-9e8b-440f-a23a-f230d0656275" (UID: "620abc13-9e8b-440f-a23a-f230d0656275"). InnerVolumeSpecName "kube-api-access-nx9nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:48.631211 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.631188 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-istio-token" pod "620abc13-9e8b-440f-a23a-f230d0656275" (UID: "620abc13-9e8b-440f-a23a-f230d0656275"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:48.631295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.631191 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-kube-api-access-jzqmh" pod "d54db83b-2f0a-46d1-90de-84c048ff4e0d" (UID: "d54db83b-2f0a-46d1-90de-84c048ff4e0d"). InnerVolumeSpecName "kube-api-access-jzqmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:48.631412 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.631397 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-token" pod "d54db83b-2f0a-46d1-90de-84c048ff4e0d" (UID: "d54db83b-2f0a-46d1-90de-84c048ff4e0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:48.631750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.631703 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-envoy" pod "d54db83b-2f0a-46d1-90de-84c048ff4e0d" (UID: "d54db83b-2f0a-46d1-90de-84c048ff4e0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:47:48.632229 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.632215 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-podinfo" pod "d54db83b-2f0a-46d1-90de-84c048ff4e0d" (UID: "d54db83b-2f0a-46d1-90de-84c048ff4e0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:47:48.729978 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.729948 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.729978 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.729974 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx9nz\" (UniqueName: \"kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-kube-api-access-nx9nz\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.729978 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.729984 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.729993 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620abc13-9e8b-440f-a23a-f230d0656275-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730008 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzqmh\" (UniqueName: \"kubernetes.io/projected/d54db83b-2f0a-46d1-90de-84c048ff4e0d-kube-api-access-jzqmh\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730021 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730032 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730040 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730049 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730057 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730067 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730074 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620abc13-9e8b-440f-a23a-f230d0656275-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730082 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730090 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620abc13-9e8b-440f-a23a-f230d0656275-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730099 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730107 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54db83b-2f0a-46d1-90de-84c048ff4e0d-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:48.730204 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:48.730115 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620abc13-9e8b-440f-a23a-f230d0656275-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:49.225936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.225854 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q"] Nov 03 07:47:49.348150 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.348116 4409 generic.go:356] "Generic (PLEG): container finished" podID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerID="5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c" exitCode=0 Nov 03 07:47:49.348343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.348174 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" Nov 03 07:47:49.348343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.348202 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" event={"ID":"d54db83b-2f0a-46d1-90de-84c048ff4e0d","Type":"ContainerDied","Data":"5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c"} Nov 03 07:47:49.348343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.348256 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6" event={"ID":"d54db83b-2f0a-46d1-90de-84c048ff4e0d","Type":"ContainerDied","Data":"fc728d6463d7e39f52897efba2499ce7fd97e403b8ffb6c3494b5ccd77ba3b68"} Nov 03 07:47:49.348343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.348286 4409 scope.go:122] "RemoveContainer" containerID="5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c" Nov 03 07:47:49.349832 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.349765 4409 generic.go:356] "Generic (PLEG): container finished" podID="620abc13-9e8b-440f-a23a-f230d0656275" containerID="308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82" exitCode=0 Nov 03 07:47:49.349832 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.349827 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" Nov 03 07:47:49.349832 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.349851 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" event={"ID":"620abc13-9e8b-440f-a23a-f230d0656275","Type":"ContainerDied","Data":"308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82"} Nov 03 07:47:49.349832 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.349894 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g" event={"ID":"620abc13-9e8b-440f-a23a-f230d0656275","Type":"ContainerDied","Data":"30071f8415a2173d39a565ee169cbfc4a3f2de75d13d6ee6a8e95bfebfa3492d"} Nov 03 07:47:49.358029 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.358015 4409 scope.go:122] "RemoveContainer" containerID="5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c" Nov 03 07:47:49.358318 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:47:49.358295 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c\": container with ID starting with 5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c not found: ID does not exist" containerID="5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c" Nov 03 07:47:49.358363 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.358321 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c"} err="failed to get container status \"5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c\": rpc error: code = NotFound desc = could not find container \"5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c\": container with ID starting with 5047edeec556ec2d994cfc60d68991f9f8f190909b9229804ef15b81ecde352c not found: ID does not exist" Nov 03 07:47:49.358363 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.358338 4409 scope.go:122] "RemoveContainer" containerID="308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82" Nov 03 07:47:49.366888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.366675 4409 scope.go:122] "RemoveContainer" containerID="308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82" Nov 03 07:47:49.367069 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:47:49.367045 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82\": container with ID starting with 308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82 not found: ID does not exist" containerID="308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82" Nov 03 07:47:49.367181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.367075 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82"} err="failed to get container status \"308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82\": rpc error: code = NotFound desc = could not find container \"308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82\": container with ID starting with 308f35b07fffae0dfc162ec506869387940ea414d0ef73d9f8061c90d88a9b82 not found: ID does not exist" Nov 03 07:47:49.517015 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.516936 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g"] Nov 03 07:47:49.568300 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.568266 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-s4f8g"] Nov 03 07:47:49.635387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.635354 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6"] Nov 03 07:47:49.642388 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:49.642364 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gp9f6"] Nov 03 07:47:50.918880 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:50.912727 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620abc13-9e8b-440f-a23a-f230d0656275" path="/var/lib/kubelet/pods/620abc13-9e8b-440f-a23a-f230d0656275/volumes" Nov 03 07:47:50.918880 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:50.913670 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" path="/var/lib/kubelet/pods/d54db83b-2f0a-46d1-90de-84c048ff4e0d/volumes" Nov 03 07:47:54.227187 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:54.227148 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" podUID="f3bc426d-4cee-46ca-b9fc-f20f463fb6aa" containerName="payload-pre-processing" containerID="cri-o://b33db81788f7f11ad927ebc54cd709bf1e24d1dd7e73c7178bca01c698eae4fe" gracePeriod=25 Nov 03 07:47:54.378820 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:54.378787 4409 generic.go:356] "Generic (PLEG): container finished" podID="f3bc426d-4cee-46ca-b9fc-f20f463fb6aa" containerID="b33db81788f7f11ad927ebc54cd709bf1e24d1dd7e73c7178bca01c698eae4fe" exitCode=0 Nov 03 07:47:54.378951 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:54.378868 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" event={"ID":"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa","Type":"ContainerDied","Data":"b33db81788f7f11ad927ebc54cd709bf1e24d1dd7e73c7178bca01c698eae4fe"} Nov 03 07:47:54.483387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:54.483314 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" Nov 03 07:47:54.588343 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:54.588321 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-kube-api-access-qtwt5\" (UniqueName: \"kubernetes.io/projected/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-kube-api-access-qtwt5\") pod \"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa\" (UID: \"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa\") " Nov 03 07:47:54.588499 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:54.588430 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-plugins-config-volume\") pod \"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa\" (UID: \"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa\") " Nov 03 07:47:54.588867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:54.588840 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-plugins-config-volume" pod "f3bc426d-4cee-46ca-b9fc-f20f463fb6aa" (UID: "f3bc426d-4cee-46ca-b9fc-f20f463fb6aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:47:54.590509 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:54.590487 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-kube-api-access-qtwt5" pod "f3bc426d-4cee-46ca-b9fc-f20f463fb6aa" (UID: "f3bc426d-4cee-46ca-b9fc-f20f463fb6aa"). InnerVolumeSpecName "kube-api-access-qtwt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:47:54.689936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:54.689908 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:54.689936 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:54.689933 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtwt5\" (UniqueName: \"kubernetes.io/projected/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa-kube-api-access-qtwt5\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:47:55.385831 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:55.385775 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" Nov 03 07:47:55.385831 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:55.385802 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q" event={"ID":"f3bc426d-4cee-46ca-b9fc-f20f463fb6aa","Type":"ContainerDied","Data":"f034aa4fe0005f2225af2dd4661d8a94f47ad0b9183e10f0de8e89df04e3490e"} Nov 03 07:47:55.386311 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:55.385874 4409 scope.go:122] "RemoveContainer" containerID="b33db81788f7f11ad927ebc54cd709bf1e24d1dd7e73c7178bca01c698eae4fe" Nov 03 07:47:55.554461 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:55.554431 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q"] Nov 03 07:47:55.603725 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:55.603689 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6606d6-d4fbc867-zv28q"] Nov 03 07:47:56.912444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:56.912414 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3bc426d-4cee-46ca-b9fc-f20f463fb6aa" path="/var/lib/kubelet/pods/f3bc426d-4cee-46ca-b9fc-f20f463fb6aa/volumes" Nov 03 07:47:57.461083 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:47:57.461054 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk"] Nov 03 07:48:00.483576 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.483543 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6"] Nov 03 07:48:00.484179 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.483774 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" podUID="471524d1-c544-492a-b04b-4ee01b5bd0e0" containerName="maas-api" containerID="cri-o://e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb" gracePeriod=30 Nov 03 07:48:00.738051 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.737999 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:48:00.856574 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.856508 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-metrics-certs\") pod \"471524d1-c544-492a-b04b-4ee01b5bd0e0\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " Nov 03 07:48:00.856832 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.856656 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/471524d1-c544-492a-b04b-4ee01b5bd0e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/471524d1-c544-492a-b04b-4ee01b5bd0e0-openshift-service-ca\") pod \"471524d1-c544-492a-b04b-4ee01b5bd0e0\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " Nov 03 07:48:00.856832 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.856697 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/471524d1-c544-492a-b04b-4ee01b5bd0e0-kube-api-access-l84sg\" (UniqueName: \"kubernetes.io/projected/471524d1-c544-492a-b04b-4ee01b5bd0e0-kube-api-access-l84sg\") pod \"471524d1-c544-492a-b04b-4ee01b5bd0e0\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " Nov 03 07:48:00.856832 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.856754 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-maas-api-tls\") pod \"471524d1-c544-492a-b04b-4ee01b5bd0e0\" (UID: \"471524d1-c544-492a-b04b-4ee01b5bd0e0\") " Nov 03 07:48:00.870859 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.857149 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/471524d1-c544-492a-b04b-4ee01b5bd0e0-openshift-service-ca" pod "471524d1-c544-492a-b04b-4ee01b5bd0e0" (UID: "471524d1-c544-492a-b04b-4ee01b5bd0e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:48:00.870859 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.857258 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/471524d1-c544-492a-b04b-4ee01b5bd0e0-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:00.870859 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.869648 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-metrics-certs" pod "471524d1-c544-492a-b04b-4ee01b5bd0e0" (UID: "471524d1-c544-492a-b04b-4ee01b5bd0e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:48:00.873963 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.873930 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471524d1-c544-492a-b04b-4ee01b5bd0e0-kube-api-access-l84sg" pod "471524d1-c544-492a-b04b-4ee01b5bd0e0" (UID: "471524d1-c544-492a-b04b-4ee01b5bd0e0"). InnerVolumeSpecName "kube-api-access-l84sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:00.876952 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.876921 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-maas-api-tls" pod "471524d1-c544-492a-b04b-4ee01b5bd0e0" (UID: "471524d1-c544-492a-b04b-4ee01b5bd0e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:48:00.957893 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.957863 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:00.957893 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.957889 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84sg\" (UniqueName: \"kubernetes.io/projected/471524d1-c544-492a-b04b-4ee01b5bd0e0-kube-api-access-l84sg\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:00.958071 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:00.957900 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/471524d1-c544-492a-b04b-4ee01b5bd0e0-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:01.426564 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:01.426531 4409 generic.go:356] "Generic (PLEG): container finished" podID="471524d1-c544-492a-b04b-4ee01b5bd0e0" containerID="e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb" exitCode=0 Nov 03 07:48:01.426771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:01.426594 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" Nov 03 07:48:01.426771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:01.426614 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" event={"ID":"471524d1-c544-492a-b04b-4ee01b5bd0e0","Type":"ContainerDied","Data":"e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb"} Nov 03 07:48:01.426771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:01.426679 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6" event={"ID":"471524d1-c544-492a-b04b-4ee01b5bd0e0","Type":"ContainerDied","Data":"d3804e35703dd74c9d6b101930d7a82d7b1913ec5f3c009d43ef35a3b30bcdad"} Nov 03 07:48:01.426771 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:01.426740 4409 scope.go:122] "RemoveContainer" containerID="e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb" Nov 03 07:48:01.447230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:01.445101 4409 scope.go:122] "RemoveContainer" containerID="e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb" Nov 03 07:48:01.447670 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:48:01.447608 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb\": container with ID starting with e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb not found: ID does not exist" containerID="e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb" Nov 03 07:48:01.447670 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:01.447657 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb"} err="failed to get container status \"e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb\": rpc error: code = NotFound desc = could not find container \"e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb\": container with ID starting with e3a50e60ca92665d825e24e14fc5c9d374428910ce5838a40829aa3b23d721cb not found: ID does not exist" Nov 03 07:48:01.595098 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:01.595069 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6"] Nov 03 07:48:01.644421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:01.644397 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50ae96-6d6748546d-l5ms6"] Nov 03 07:48:02.461901 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:02.461862 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" podUID="eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456" containerName="payload-pre-processing" containerID="cri-o://d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332" gracePeriod=25 Nov 03 07:48:02.911437 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:02.911404 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="471524d1-c544-492a-b04b-4ee01b5bd0e0" path="/var/lib/kubelet/pods/471524d1-c544-492a-b04b-4ee01b5bd0e0/volumes" Nov 03 07:48:05.219231 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:05.219185 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" podUID="eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 07:48:08.774137 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:48:08.774097 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:48:11.099418 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:11.099384 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f"] Nov 03 07:48:11.099875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:11.099583 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" containerID="cri-o://bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131" gracePeriod=30 Nov 03 07:48:14.083934 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:14.083901 4409 patch_prober.go:29] interesting pod/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 07:48:14.084298 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:14.083960 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 07:48:15.218783 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:15.218744 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" podUID="eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 07:48:16.355686 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.355647 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:48:16.438635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.438567 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-certs\") pod \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " Nov 03 07:48:16.438635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.438605 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-envoy\") pod \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " Nov 03 07:48:16.438896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.438634 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-data\") pod \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " Nov 03 07:48:16.438896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.438780 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-credential-socket\") pod \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " Nov 03 07:48:16.438896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.438816 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-token\" (UniqueName: \"kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-token\") pod \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " Nov 03 07:48:16.438896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.438846 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istiod-ca-cert\") pod \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " Nov 03 07:48:16.438896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.438863 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-certs" pod "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" (UID: "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:16.438896 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.438883 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-podinfo\") pod \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " Nov 03 07:48:16.439219 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.438945 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-kube-api-access-rglcf\" (UniqueName: \"kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-kube-api-access-rglcf\") pod \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " Nov 03 07:48:16.439219 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.438979 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-data" pod "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" (UID: "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:16.439219 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.438988 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-socket\") pod \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\" (UID: \"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4\") " Nov 03 07:48:16.439219 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.439110 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-credential-socket" pod "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" (UID: "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:16.439219 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.439197 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-socket" pod "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" (UID: "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:16.439468 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.439241 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istiod-ca-cert" pod "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" (UID: "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:48:16.439525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.439464 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:16.439525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.439485 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:16.439525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.439499 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:16.439525 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.439516 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:16.439923 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.439534 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:16.440972 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.440942 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-envoy" pod "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" (UID: "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:16.441104 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.441053 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-token" pod "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" (UID: "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:16.441199 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.441178 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-kube-api-access-rglcf" pod "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" (UID: "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4"). InnerVolumeSpecName "kube-api-access-rglcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:16.441365 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.441339 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-podinfo" pod "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" (UID: "7591fd6e-392a-4aa0-8b3f-2ba98cc00db4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:48:16.511821 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.511790 4409 generic.go:356] "Generic (PLEG): container finished" podID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerID="bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131" exitCode=0 Nov 03 07:48:16.511973 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.511860 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" event={"ID":"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4","Type":"ContainerDied","Data":"bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131"} Nov 03 07:48:16.511973 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.511866 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" Nov 03 07:48:16.511973 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.511915 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f" event={"ID":"7591fd6e-392a-4aa0-8b3f-2ba98cc00db4","Type":"ContainerDied","Data":"87bbcc5f9d8884b50b9abdfc728628df99f2341a1c1e3d9f8b60f6541baf28dc"} Nov 03 07:48:16.511973 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.511939 4409 scope.go:122] "RemoveContainer" containerID="bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131" Nov 03 07:48:16.521638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.521622 4409 scope.go:122] "RemoveContainer" containerID="8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa" Nov 03 07:48:16.530458 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.530428 4409 scope.go:122] "RemoveContainer" containerID="bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131" Nov 03 07:48:16.530824 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:48:16.530799 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131\": container with ID starting with bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131 not found: ID does not exist" containerID="bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131" Nov 03 07:48:16.530906 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.530836 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131"} err="failed to get container status \"bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131\": rpc error: code = NotFound desc = could not find container \"bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131\": container with ID starting with bdb2384afc642af13b3328a93cc06147c1751bdff7f69dc7d9b04f6a2e7ee131 not found: ID does not exist" Nov 03 07:48:16.530906 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.530862 4409 scope.go:122] "RemoveContainer" containerID="8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa" Nov 03 07:48:16.531180 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:48:16.531144 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa\": container with ID starting with 8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa not found: ID does not exist" containerID="8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa" Nov 03 07:48:16.531272 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.531185 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa"} err="failed to get container status \"8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa\": rpc error: code = NotFound desc = could not find container \"8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa\": container with ID starting with 8ab0cabc59903e3670a9b3b59234f1a8ce6c5b5e1c5669ce4040975a5e1026aa not found: ID does not exist" Nov 03 07:48:16.540707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.540684 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:16.540707 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.540708 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:16.540867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.540738 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:16.540867 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.540751 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rglcf\" (UniqueName: \"kubernetes.io/projected/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4-kube-api-access-rglcf\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:16.666553 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.666525 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f"] Nov 03 07:48:16.672021 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.672000 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-50ae96-openshift-default-65c7ddfc5f-d826f"] Nov 03 07:48:16.911929 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:16.911893 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" path="/var/lib/kubelet/pods/7591fd6e-392a-4aa0-8b3f-2ba98cc00db4/volumes" Nov 03 07:48:22.712444 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:22.712418 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" Nov 03 07:48:22.803762 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:22.803737 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-kube-api-access-ch2pc\" (UniqueName: \"kubernetes.io/projected/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-kube-api-access-ch2pc\") pod \"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456\" (UID: \"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456\") " Nov 03 07:48:22.803888 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:22.803841 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-plugins-config-volume\") pod \"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456\" (UID: \"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456\") " Nov 03 07:48:22.804172 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:22.804150 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-plugins-config-volume" pod "eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456" (UID: "eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:48:22.805934 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:22.805912 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-kube-api-access-ch2pc" pod "eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456" (UID: "eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456"). InnerVolumeSpecName "kube-api-access-ch2pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:22.905248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:22.905217 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:22.905248 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:22.905242 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch2pc\" (UniqueName: \"kubernetes.io/projected/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456-kube-api-access-ch2pc\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:23.553486 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:23.553452 4409 generic.go:356] "Generic (PLEG): container finished" podID="eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456" containerID="d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332" exitCode=0 Nov 03 07:48:23.553668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:23.553500 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" event={"ID":"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456","Type":"ContainerDied","Data":"d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332"} Nov 03 07:48:23.553668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:23.553511 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" Nov 03 07:48:23.553668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:23.553538 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk" event={"ID":"eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456","Type":"ContainerDied","Data":"59adeb4f5909379f609735be9c5746525b55078ee9ad800851168536878ddfea"} Nov 03 07:48:23.553668 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:23.553560 4409 scope.go:122] "RemoveContainer" containerID="d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332" Nov 03 07:48:23.562658 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:23.562640 4409 scope.go:122] "RemoveContainer" containerID="d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332" Nov 03 07:48:23.562997 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:48:23.562980 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332\": container with ID starting with d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332 not found: ID does not exist" containerID="d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332" Nov 03 07:48:23.563054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:23.563007 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332"} err="failed to get container status \"d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332\": rpc error: code = NotFound desc = could not find container \"d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332\": container with ID starting with d5ffa75bc7d3fe06b5d9222e73d701a93719be2ef55ceac04eb4dda31f042332 not found: ID does not exist" Nov 03 07:48:23.723487 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:23.723455 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk"] Nov 03 07:48:23.774329 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:23.774298 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50ae96-fcf4b47-78qgk"] Nov 03 07:48:24.057838 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.057798 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn"] Nov 03 07:48:24.058032 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.057994 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" podUID="18c3e73a-ca38-428f-b5ab-09b864ca5477" containerName="maas-api" containerID="cri-o://ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5" gracePeriod=30 Nov 03 07:48:24.313137 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.313078 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:48:24.420949 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.420923 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c3e73a-ca38-428f-b5ab-09b864ca5477-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18c3e73a-ca38-428f-b5ab-09b864ca5477-openshift-service-ca\") pod \"18c3e73a-ca38-428f-b5ab-09b864ca5477\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " Nov 03 07:48:24.421082 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.420960 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-maas-api-tls\") pod \"18c3e73a-ca38-428f-b5ab-09b864ca5477\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " Nov 03 07:48:24.421123 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.421088 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c3e73a-ca38-428f-b5ab-09b864ca5477-kube-api-access-pr46f\" (UniqueName: \"kubernetes.io/projected/18c3e73a-ca38-428f-b5ab-09b864ca5477-kube-api-access-pr46f\") pod \"18c3e73a-ca38-428f-b5ab-09b864ca5477\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " Nov 03 07:48:24.421171 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.421153 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-metrics-certs\") pod \"18c3e73a-ca38-428f-b5ab-09b864ca5477\" (UID: \"18c3e73a-ca38-428f-b5ab-09b864ca5477\") " Nov 03 07:48:24.421322 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.421303 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c3e73a-ca38-428f-b5ab-09b864ca5477-openshift-service-ca" pod "18c3e73a-ca38-428f-b5ab-09b864ca5477" (UID: "18c3e73a-ca38-428f-b5ab-09b864ca5477"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:48:24.421459 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.421437 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18c3e73a-ca38-428f-b5ab-09b864ca5477-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:24.423269 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.423243 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-metrics-certs" pod "18c3e73a-ca38-428f-b5ab-09b864ca5477" (UID: "18c3e73a-ca38-428f-b5ab-09b864ca5477"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:48:24.423359 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.423266 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-maas-api-tls" pod "18c3e73a-ca38-428f-b5ab-09b864ca5477" (UID: "18c3e73a-ca38-428f-b5ab-09b864ca5477"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:48:24.423359 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.423269 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c3e73a-ca38-428f-b5ab-09b864ca5477-kube-api-access-pr46f" pod "18c3e73a-ca38-428f-b5ab-09b864ca5477" (UID: "18c3e73a-ca38-428f-b5ab-09b864ca5477"). InnerVolumeSpecName "kube-api-access-pr46f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:24.522864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.522832 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr46f\" (UniqueName: \"kubernetes.io/projected/18c3e73a-ca38-428f-b5ab-09b864ca5477-kube-api-access-pr46f\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:24.522864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.522858 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:24.522864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.522867 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18c3e73a-ca38-428f-b5ab-09b864ca5477-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:24.569182 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.568898 4409 generic.go:356] "Generic (PLEG): container finished" podID="18c3e73a-ca38-428f-b5ab-09b864ca5477" containerID="ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5" exitCode=0 Nov 03 07:48:24.569182 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.568945 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" Nov 03 07:48:24.569182 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.568977 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" event={"ID":"18c3e73a-ca38-428f-b5ab-09b864ca5477","Type":"ContainerDied","Data":"ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5"} Nov 03 07:48:24.569182 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.569032 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn" event={"ID":"18c3e73a-ca38-428f-b5ab-09b864ca5477","Type":"ContainerDied","Data":"4c8f26904de5ffa5a2589d890103f19fb65509da24de9cec6a682a9bbea60485"} Nov 03 07:48:24.569182 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.569053 4409 scope.go:122] "RemoveContainer" containerID="ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5" Nov 03 07:48:24.584289 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.584269 4409 scope.go:122] "RemoveContainer" containerID="ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5" Nov 03 07:48:24.584694 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:48:24.584673 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5\": container with ID starting with ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5 not found: ID does not exist" containerID="ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5" Nov 03 07:48:24.584785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.584724 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5"} err="failed to get container status \"ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5\": rpc error: code = NotFound desc = could not find container \"ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5\": container with ID starting with ce28d3cf22273fc2b5e8230b1c24c393b8f2a0389ceea4155d0f454571e1bfd5 not found: ID does not exist" Nov 03 07:48:24.737021 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.736998 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn"] Nov 03 07:48:24.743619 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.743595 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eac348-57d4cbd54c-52gtn"] Nov 03 07:48:24.917585 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.917518 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c3e73a-ca38-428f-b5ab-09b864ca5477" path="/var/lib/kubelet/pods/18c3e73a-ca38-428f-b5ab-09b864ca5477/volumes" Nov 03 07:48:24.917944 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:24.917929 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456" path="/var/lib/kubelet/pods/eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456/volumes" Nov 03 07:48:36.615966 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:36.615925 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z"] Nov 03 07:48:36.616498 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:36.616143 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerName="istio-proxy" containerID="cri-o://e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548" gracePeriod=30 Nov 03 07:48:38.774553 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:48:38.774510 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:48:41.879594 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.879567 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:48:41.891642 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.891583 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-socket\") pod \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " Nov 03 07:48:41.891642 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.891614 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istiod-ca-cert\") pod \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " Nov 03 07:48:41.891856 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.891671 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-credential-socket\") pod \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " Nov 03 07:48:41.891856 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.891691 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-envoy\") pod \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " Nov 03 07:48:41.891856 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.891771 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-podinfo\") pod \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " Nov 03 07:48:41.891856 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.891804 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-token\" (UniqueName: \"kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-token\") pod \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " Nov 03 07:48:41.891856 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.891847 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-data\") pod \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " Nov 03 07:48:41.892123 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.891869 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-kube-api-access-lhkv4\" (UniqueName: \"kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-kube-api-access-lhkv4\") pod \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " Nov 03 07:48:41.892123 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.891890 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-certs\") pod \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\" (UID: \"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed\") " Nov 03 07:48:41.892123 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.891985 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-socket" pod "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" (UID: "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:41.892123 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.892001 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-credential-socket" pod "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" (UID: "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:41.892345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.892227 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-data" pod "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" (UID: "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:41.892345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.892168 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istiod-ca-cert" pod "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" (UID: "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:48:41.892345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.892308 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:41.892345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.892327 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:41.892345 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.892341 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:41.892604 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.892356 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:41.892604 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.892427 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-certs" pod "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" (UID: "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:41.894239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.894213 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-podinfo" pod "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" (UID: "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:48:41.894386 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.894244 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-envoy" pod "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" (UID: "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:48:41.894434 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.894414 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-kube-api-access-lhkv4" pod "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" (UID: "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed"). InnerVolumeSpecName "kube-api-access-lhkv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:41.894672 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.894643 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-token" pod "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" (UID: "aa1ee478-5863-4f95-8512-d6a2f5c4f6ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:41.993165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.993125 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:41.993165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.993155 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:41.993165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.993165 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:41.993165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.993175 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhkv4\" (UniqueName: \"kubernetes.io/projected/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-kube-api-access-lhkv4\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:41.993453 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:41.993185 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:42.661817 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.661783 4409 generic.go:356] "Generic (PLEG): container finished" podID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerID="e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548" exitCode=0 Nov 03 07:48:42.662026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.661858 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" Nov 03 07:48:42.662026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.661876 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" event={"ID":"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed","Type":"ContainerDied","Data":"e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548"} Nov 03 07:48:42.662026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.661958 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z" event={"ID":"aa1ee478-5863-4f95-8512-d6a2f5c4f6ed","Type":"ContainerDied","Data":"2f543dc415afff22f342bdca3124562f5c95d5f38cf171ba8d27dbd2402ecf3a"} Nov 03 07:48:42.662026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.661983 4409 scope.go:122] "RemoveContainer" containerID="e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548" Nov 03 07:48:42.671651 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.671633 4409 scope.go:122] "RemoveContainer" containerID="1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1" Nov 03 07:48:42.679890 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.679870 4409 scope.go:122] "RemoveContainer" containerID="e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548" Nov 03 07:48:42.680160 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:48:42.680141 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548\": container with ID starting with e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548 not found: ID does not exist" containerID="e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548" Nov 03 07:48:42.680212 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.680170 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548"} err="failed to get container status \"e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548\": rpc error: code = NotFound desc = could not find container \"e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548\": container with ID starting with e548145b897b04f7a706a05b7d47841bc7b5e827f4d476a2f14a2e6ebbafb548 not found: ID does not exist" Nov 03 07:48:42.680212 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.680187 4409 scope.go:122] "RemoveContainer" containerID="1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1" Nov 03 07:48:42.680420 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:48:42.680402 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1\": container with ID starting with 1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1 not found: ID does not exist" containerID="1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1" Nov 03 07:48:42.680462 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.680428 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1"} err="failed to get container status \"1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1\": rpc error: code = NotFound desc = could not find container \"1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1\": container with ID starting with 1b1c96a8db756a7c9415a2a85d112313e3ec2b18980b3f9c0159a5a90e000bf1 not found: ID does not exist" Nov 03 07:48:42.830111 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.830074 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z"] Nov 03 07:48:42.879349 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.879313 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-eac348-openshift-default-7ff86ddc97-98s5z"] Nov 03 07:48:42.911352 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:42.911318 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" path="/var/lib/kubelet/pods/aa1ee478-5863-4f95-8512-d6a2f5c4f6ed/volumes" Nov 03 07:48:48.703240 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:48.703204 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6"] Nov 03 07:48:53.703689 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:53.703624 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" podUID="963a8556-7780-49ec-84bf-de9a09ff4065" containerName="payload-pre-processing" containerID="cri-o://153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d" gracePeriod=25 Nov 03 07:48:53.956554 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:53.956496 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" Nov 03 07:48:54.006002 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.005968 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963a8556-7780-49ec-84bf-de9a09ff4065-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963a8556-7780-49ec-84bf-de9a09ff4065-plugins-config-volume\") pod \"963a8556-7780-49ec-84bf-de9a09ff4065\" (UID: \"963a8556-7780-49ec-84bf-de9a09ff4065\") " Nov 03 07:48:54.006148 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.006029 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963a8556-7780-49ec-84bf-de9a09ff4065-kube-api-access-tvwrw\" (UniqueName: \"kubernetes.io/projected/963a8556-7780-49ec-84bf-de9a09ff4065-kube-api-access-tvwrw\") pod \"963a8556-7780-49ec-84bf-de9a09ff4065\" (UID: \"963a8556-7780-49ec-84bf-de9a09ff4065\") " Nov 03 07:48:54.006362 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.006338 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963a8556-7780-49ec-84bf-de9a09ff4065-plugins-config-volume" pod "963a8556-7780-49ec-84bf-de9a09ff4065" (UID: "963a8556-7780-49ec-84bf-de9a09ff4065"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:48:54.008223 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.008194 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963a8556-7780-49ec-84bf-de9a09ff4065-kube-api-access-tvwrw" pod "963a8556-7780-49ec-84bf-de9a09ff4065" (UID: "963a8556-7780-49ec-84bf-de9a09ff4065"). InnerVolumeSpecName "kube-api-access-tvwrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:48:54.107623 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.107590 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963a8556-7780-49ec-84bf-de9a09ff4065-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:54.107623 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.107619 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvwrw\" (UniqueName: \"kubernetes.io/projected/963a8556-7780-49ec-84bf-de9a09ff4065-kube-api-access-tvwrw\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:48:54.721610 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.721574 4409 generic.go:356] "Generic (PLEG): container finished" podID="963a8556-7780-49ec-84bf-de9a09ff4065" containerID="153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d" exitCode=0 Nov 03 07:48:54.722128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.721613 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" Nov 03 07:48:54.722128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.721621 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" event={"ID":"963a8556-7780-49ec-84bf-de9a09ff4065","Type":"ContainerDied","Data":"153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d"} Nov 03 07:48:54.722128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.721698 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6" event={"ID":"963a8556-7780-49ec-84bf-de9a09ff4065","Type":"ContainerDied","Data":"fb220c2c8147e3f5b3d5829393b0c7191f35af22e74a216d75dabf4a465d99c0"} Nov 03 07:48:54.722128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.721746 4409 scope.go:122] "RemoveContainer" containerID="153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d" Nov 03 07:48:54.731132 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.731115 4409 scope.go:122] "RemoveContainer" containerID="153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d" Nov 03 07:48:54.731402 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:48:54.731379 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d\": container with ID starting with 153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d not found: ID does not exist" containerID="153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d" Nov 03 07:48:54.731442 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.731409 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d"} err="failed to get container status \"153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d\": rpc error: code = NotFound desc = could not find container \"153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d\": container with ID starting with 153e3dc89a34b531bdea2a765a401cce14f666d2ff5870609d51dd318c475d6d not found: ID does not exist" Nov 03 07:48:54.890153 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.890125 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6"] Nov 03 07:48:54.939388 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:54.939361 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ffee69-757d59959f-9nqt6"] Nov 03 07:48:56.910807 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:48:56.910765 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963a8556-7780-49ec-84bf-de9a09ff4065" path="/var/lib/kubelet/pods/963a8556-7780-49ec-84bf-de9a09ff4065/volumes" Nov 03 07:49:05.375294 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:05.375267 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:49:05.375784 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:05.375505 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:49:08.774343 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:49:08.774302 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:49:11.680035 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:11.679999 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv"] Nov 03 07:49:11.680491 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:11.680244 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" podUID="3e6e68f6-d602-4c83-80a1-0323c92a6f98" containerName="maas-api" containerID="cri-o://aaf825570abf3c7da667c2b84efe6db4c3343fd3c54311a7b0c2994004e52eb6" gracePeriod=30 Nov 03 07:49:11.805601 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:11.805564 4409 generic.go:356] "Generic (PLEG): container finished" podID="3e6e68f6-d602-4c83-80a1-0323c92a6f98" containerID="aaf825570abf3c7da667c2b84efe6db4c3343fd3c54311a7b0c2994004e52eb6" exitCode=0 Nov 03 07:49:11.805751 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:11.805629 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" event={"ID":"3e6e68f6-d602-4c83-80a1-0323c92a6f98","Type":"ContainerDied","Data":"aaf825570abf3c7da667c2b84efe6db4c3343fd3c54311a7b0c2994004e52eb6"} Nov 03 07:49:11.930132 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:11.930056 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:49:12.075337 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.075309 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-maas-api-tls\") pod \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " Nov 03 07:49:12.075522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.075385 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6e68f6-d602-4c83-80a1-0323c92a6f98-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e68f6-d602-4c83-80a1-0323c92a6f98-openshift-service-ca\") pod \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " Nov 03 07:49:12.075522 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.075485 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-metrics-certs\") pod \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " Nov 03 07:49:12.075664 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.075539 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6e68f6-d602-4c83-80a1-0323c92a6f98-kube-api-access-vbq99\" (UniqueName: \"kubernetes.io/projected/3e6e68f6-d602-4c83-80a1-0323c92a6f98-kube-api-access-vbq99\") pod \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\" (UID: \"3e6e68f6-d602-4c83-80a1-0323c92a6f98\") " Nov 03 07:49:12.075792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.075759 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6e68f6-d602-4c83-80a1-0323c92a6f98-openshift-service-ca" pod "3e6e68f6-d602-4c83-80a1-0323c92a6f98" (UID: "3e6e68f6-d602-4c83-80a1-0323c92a6f98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:49:12.075922 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.075904 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e68f6-d602-4c83-80a1-0323c92a6f98-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:12.077756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.077702 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-maas-api-tls" pod "3e6e68f6-d602-4c83-80a1-0323c92a6f98" (UID: "3e6e68f6-d602-4c83-80a1-0323c92a6f98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:49:12.077899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.077763 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-metrics-certs" pod "3e6e68f6-d602-4c83-80a1-0323c92a6f98" (UID: "3e6e68f6-d602-4c83-80a1-0323c92a6f98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:49:12.077899 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.077869 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6e68f6-d602-4c83-80a1-0323c92a6f98-kube-api-access-vbq99" pod "3e6e68f6-d602-4c83-80a1-0323c92a6f98" (UID: "3e6e68f6-d602-4c83-80a1-0323c92a6f98"). InnerVolumeSpecName "kube-api-access-vbq99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:49:12.177000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.176972 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:12.177000 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.177002 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6e68f6-d602-4c83-80a1-0323c92a6f98-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:12.177169 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.177012 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbq99\" (UniqueName: \"kubernetes.io/projected/3e6e68f6-d602-4c83-80a1-0323c92a6f98-kube-api-access-vbq99\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:12.813028 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.812992 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" Nov 03 07:49:12.813548 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.812992 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv" event={"ID":"3e6e68f6-d602-4c83-80a1-0323c92a6f98","Type":"ContainerDied","Data":"9f73bc06a7e2c9b0396d40d91bcd1615a5e8564585904d381dce25e1a75131ba"} Nov 03 07:49:12.813548 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.813145 4409 scope.go:122] "RemoveContainer" containerID="aaf825570abf3c7da667c2b84efe6db4c3343fd3c54311a7b0c2994004e52eb6" Nov 03 07:49:12.980833 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:12.980795 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv"] Nov 03 07:49:13.030050 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:13.030019 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ffee69-bd5f9cdc4-twbmv"] Nov 03 07:49:14.912275 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:14.912245 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6e68f6-d602-4c83-80a1-0323c92a6f98" path="/var/lib/kubelet/pods/3e6e68f6-d602-4c83-80a1-0323c92a6f98/volumes" Nov 03 07:49:23.037425 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:23.037208 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z"] Nov 03 07:49:23.038176 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:23.038147 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" podUID="f6dbc02a-e589-4172-8071-272bd1a343d4" containerName="istio-proxy" containerID="cri-o://c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4" gracePeriod=30 Nov 03 07:49:28.285295 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.285268 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:49:28.317484 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.317397 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-socket\") pod \"f6dbc02a-e589-4172-8071-272bd1a343d4\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " Nov 03 07:49:28.317484 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.317437 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-envoy\") pod \"f6dbc02a-e589-4172-8071-272bd1a343d4\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " Nov 03 07:49:28.317484 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.317470 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-token\" (UniqueName: \"kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-token\") pod \"f6dbc02a-e589-4172-8071-272bd1a343d4\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " Nov 03 07:49:28.317484 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.317488 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-credential-socket\") pod \"f6dbc02a-e589-4172-8071-272bd1a343d4\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " Nov 03 07:49:28.317860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.317515 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6dbc02a-e589-4172-8071-272bd1a343d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6dbc02a-e589-4172-8071-272bd1a343d4-istiod-ca-cert\") pod \"f6dbc02a-e589-4172-8071-272bd1a343d4\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " Nov 03 07:49:28.317860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.317535 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-podinfo\") pod \"f6dbc02a-e589-4172-8071-272bd1a343d4\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " Nov 03 07:49:28.317860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.317627 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-certs\") pod \"f6dbc02a-e589-4172-8071-272bd1a343d4\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " Nov 03 07:49:28.317860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.317649 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-kube-api-access-6pv5t\" (UniqueName: \"kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-kube-api-access-6pv5t\") pod \"f6dbc02a-e589-4172-8071-272bd1a343d4\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " Nov 03 07:49:28.317860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.317669 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-data\") pod \"f6dbc02a-e589-4172-8071-272bd1a343d4\" (UID: \"f6dbc02a-e589-4172-8071-272bd1a343d4\") " Nov 03 07:49:28.318128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.317958 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-socket" pod "f6dbc02a-e589-4172-8071-272bd1a343d4" (UID: "f6dbc02a-e589-4172-8071-272bd1a343d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:49:28.318128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.318061 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6dbc02a-e589-4172-8071-272bd1a343d4-istiod-ca-cert" pod "f6dbc02a-e589-4172-8071-272bd1a343d4" (UID: "f6dbc02a-e589-4172-8071-272bd1a343d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:49:28.318256 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.318236 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-data" pod "f6dbc02a-e589-4172-8071-272bd1a343d4" (UID: "f6dbc02a-e589-4172-8071-272bd1a343d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:49:28.318469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.318313 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:28.318469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.318340 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6dbc02a-e589-4172-8071-272bd1a343d4-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:28.318469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.318342 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-certs" pod "f6dbc02a-e589-4172-8071-272bd1a343d4" (UID: "f6dbc02a-e589-4172-8071-272bd1a343d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:49:28.318469 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.318434 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-credential-socket" pod "f6dbc02a-e589-4172-8071-272bd1a343d4" (UID: "f6dbc02a-e589-4172-8071-272bd1a343d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:49:28.320094 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.320063 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-envoy" pod "f6dbc02a-e589-4172-8071-272bd1a343d4" (UID: "f6dbc02a-e589-4172-8071-272bd1a343d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:49:28.320187 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.320085 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-token" pod "f6dbc02a-e589-4172-8071-272bd1a343d4" (UID: "f6dbc02a-e589-4172-8071-272bd1a343d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:49:28.320187 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.320121 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-podinfo" pod "f6dbc02a-e589-4172-8071-272bd1a343d4" (UID: "f6dbc02a-e589-4172-8071-272bd1a343d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:49:28.320266 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.320216 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-kube-api-access-6pv5t" pod "f6dbc02a-e589-4172-8071-272bd1a343d4" (UID: "f6dbc02a-e589-4172-8071-272bd1a343d4"). InnerVolumeSpecName "kube-api-access-6pv5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:49:28.419476 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.419432 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:28.419476 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.419466 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:28.419476 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.419477 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:28.419476 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.419486 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:28.419802 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.419495 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pv5t\" (UniqueName: \"kubernetes.io/projected/f6dbc02a-e589-4172-8071-272bd1a343d4-kube-api-access-6pv5t\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:28.419802 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.419505 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:28.419802 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.419515 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6dbc02a-e589-4172-8071-272bd1a343d4-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:49:28.888563 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.888532 4409 generic.go:356] "Generic (PLEG): container finished" podID="f6dbc02a-e589-4172-8071-272bd1a343d4" containerID="c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4" exitCode=0 Nov 03 07:49:28.888785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.888587 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" Nov 03 07:49:28.888785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.888616 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" event={"ID":"f6dbc02a-e589-4172-8071-272bd1a343d4","Type":"ContainerDied","Data":"c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4"} Nov 03 07:49:28.888785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.888674 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z" event={"ID":"f6dbc02a-e589-4172-8071-272bd1a343d4","Type":"ContainerDied","Data":"ff3302dab3135cb846639564bc9af13333ea077125adea78d456ed92d0c37924"} Nov 03 07:49:28.888785 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.888698 4409 scope.go:122] "RemoveContainer" containerID="c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4" Nov 03 07:49:28.898925 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.898901 4409 scope.go:122] "RemoveContainer" containerID="c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4" Nov 03 07:49:28.899244 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:49:28.899224 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4\": container with ID starting with c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4 not found: ID does not exist" containerID="c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4" Nov 03 07:49:28.899328 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:28.899251 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4"} err="failed to get container status \"c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4\": rpc error: code = NotFound desc = could not find container \"c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4\": container with ID starting with c3501c39e48df8c18db1fafcc6b8669b434b02e3aa79b858a73ae502470aadf4 not found: ID does not exist" Nov 03 07:49:29.067905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:29.067879 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z"] Nov 03 07:49:29.117233 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:29.117208 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ffee69-openshift-default-86bcc884c-fhk4z"] Nov 03 07:49:30.051253 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051220 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d848cbdf4-g6tw2"] Nov 03 07:49:30.051638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051606 4409 cpu_manager.go:414] "containerMap: removing container" podUID="53c50c34-72c7-4e1b-a926-4e9ef47b9fd5" containerName="payload-pre-processing" Nov 03 07:49:30.051638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051618 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c50c34-72c7-4e1b-a926-4e9ef47b9fd5" containerName="payload-pre-processing" Nov 03 07:49:30.051638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051633 4409 cpu_manager.go:414] "containerMap: removing container" podUID="3e6e68f6-d602-4c83-80a1-0323c92a6f98" containerName="maas-api" Nov 03 07:49:30.051638 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051639 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6e68f6-d602-4c83-80a1-0323c92a6f98" containerName="maas-api" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051646 4409 cpu_manager.go:414] "containerMap: removing container" podUID="d1098b16-19e9-48a1-8a4a-507ab6b616de" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051652 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1098b16-19e9-48a1-8a4a-507ab6b616de" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051664 4409 cpu_manager.go:414] "containerMap: removing container" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051669 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051677 4409 cpu_manager.go:414] "containerMap: removing container" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051683 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051695 4409 cpu_manager.go:414] "containerMap: removing container" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051700 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051707 4409 cpu_manager.go:414] "containerMap: removing container" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051726 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051732 4409 cpu_manager.go:414] "containerMap: removing container" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051737 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051745 4409 cpu_manager.go:414] "containerMap: removing container" podUID="963a8556-7780-49ec-84bf-de9a09ff4065" containerName="payload-pre-processing" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051749 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963a8556-7780-49ec-84bf-de9a09ff4065" containerName="payload-pre-processing" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051755 4409 cpu_manager.go:414] "containerMap: removing container" podUID="471524d1-c544-492a-b04b-4ee01b5bd0e0" containerName="maas-api" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051759 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="471524d1-c544-492a-b04b-4ee01b5bd0e0" containerName="maas-api" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051767 4409 cpu_manager.go:414] "containerMap: removing container" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051771 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051779 4409 cpu_manager.go:414] "containerMap: removing container" podUID="18c3e73a-ca38-428f-b5ab-09b864ca5477" containerName="maas-api" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051783 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c3e73a-ca38-428f-b5ab-09b864ca5477" containerName="maas-api" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051791 4409 cpu_manager.go:414] "containerMap: removing container" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051795 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051803 4409 cpu_manager.go:414] "containerMap: removing container" podUID="f3bc426d-4cee-46ca-b9fc-f20f463fb6aa" containerName="payload-pre-processing" Nov 03 07:49:30.051799 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051808 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3bc426d-4cee-46ca-b9fc-f20f463fb6aa" containerName="payload-pre-processing" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051815 4409 cpu_manager.go:414] "containerMap: removing container" podUID="f6dbc02a-e589-4172-8071-272bd1a343d4" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051820 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6dbc02a-e589-4172-8071-272bd1a343d4" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051831 4409 cpu_manager.go:414] "containerMap: removing container" podUID="79c28244-4715-46d9-9162-aeaa1d92dc1e" containerName="maas-api" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051835 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c28244-4715-46d9-9162-aeaa1d92dc1e" containerName="maas-api" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051841 4409 cpu_manager.go:414] "containerMap: removing container" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051845 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051851 4409 cpu_manager.go:414] "containerMap: removing container" podUID="34261966-3f63-47b2-9bff-e687beb22e3e" containerName="payload-pre-processing" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051855 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34261966-3f63-47b2-9bff-e687beb22e3e" containerName="payload-pre-processing" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051862 4409 cpu_manager.go:414] "containerMap: removing container" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051866 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051872 4409 cpu_manager.go:414] "containerMap: removing container" podUID="24e61dd8-0436-41bd-b80e-bdfd1f70b16b" containerName="maas-api" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051876 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e61dd8-0436-41bd-b80e-bdfd1f70b16b" containerName="maas-api" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051882 4409 cpu_manager.go:414] "containerMap: removing container" podUID="eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456" containerName="payload-pre-processing" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051887 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456" containerName="payload-pre-processing" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051938 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051946 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051953 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="620abc13-9e8b-440f-a23a-f230d0656275" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051958 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051964 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="963a8556-7780-49ec-84bf-de9a09ff4065" containerName="payload-pre-processing" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051969 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="471524d1-c544-492a-b04b-4ee01b5bd0e0" containerName="maas-api" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051975 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="25689a41-3fdd-401f-a99c-d59679618916" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051981 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c3e73a-ca38-428f-b5ab-09b864ca5477" containerName="maas-api" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051987 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="8158bcb5-641e-40c7-9ac0-eb2d59f05235" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051993 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3bc426d-4cee-46ca-b9fc-f20f463fb6aa" containerName="payload-pre-processing" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.051998 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6dbc02a-e589-4172-8071-272bd1a343d4" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.052004 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c28244-4715-46d9-9162-aeaa1d92dc1e" containerName="maas-api" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.052009 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="d816bfac-359f-463e-81b9-accd32ce79e1" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.052014 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="34261966-3f63-47b2-9bff-e687beb22e3e" containerName="payload-pre-processing" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.052019 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54db83b-2f0a-46d1-90de-84c048ff4e0d" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.052023 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e61dd8-0436-41bd-b80e-bdfd1f70b16b" containerName="maas-api" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.052029 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="eecf4c8c-f44c-4cb7-9c7a-2a76f6d5a456" containerName="payload-pre-processing" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.052035 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c50c34-72c7-4e1b-a926-4e9ef47b9fd5" containerName="payload-pre-processing" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.052041 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6e68f6-d602-4c83-80a1-0323c92a6f98" containerName="maas-api" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.052046 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1098b16-19e9-48a1-8a4a-507ab6b616de" containerName="istio-proxy" Nov 03 07:49:30.052635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.052052 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="7591fd6e-392a-4aa0-8b3f-2ba98cc00db4" containerName="istio-proxy" Nov 03 07:49:30.056466 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.056448 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:30.135428 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.135401 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz8g8\" (UniqueName: \"kubernetes.io/projected/5d310f0a-e80a-40e8-8092-dfa4518a077a-kube-api-access-dz8g8\") pod \"maas-controller-d848cbdf4-g6tw2\" (UID: \"5d310f0a-e80a-40e8-8092-dfa4518a077a\") " pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:30.135549 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.135434 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-metrics-certs\") pod \"maas-controller-d848cbdf4-g6tw2\" (UID: \"5d310f0a-e80a-40e8-8092-dfa4518a077a\") " pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:30.135549 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.135466 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-webhook-cert\") pod \"maas-controller-d848cbdf4-g6tw2\" (UID: \"5d310f0a-e80a-40e8-8092-dfa4518a077a\") " pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:30.159063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.159041 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7pc74\"" Nov 03 07:49:30.159302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.159187 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 07:49:30.159302 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.159251 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 07:49:30.159468 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.159307 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 07:49:30.162074 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.162051 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d848cbdf4-g6tw2"] Nov 03 07:49:30.236926 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.236899 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-webhook-cert\") pod \"maas-controller-d848cbdf4-g6tw2\" (UID: \"5d310f0a-e80a-40e8-8092-dfa4518a077a\") " pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:30.237076 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.237000 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz8g8\" (UniqueName: \"kubernetes.io/projected/5d310f0a-e80a-40e8-8092-dfa4518a077a-kube-api-access-dz8g8\") pod \"maas-controller-d848cbdf4-g6tw2\" (UID: \"5d310f0a-e80a-40e8-8092-dfa4518a077a\") " pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:30.237076 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.237028 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-metrics-certs\") pod \"maas-controller-d848cbdf4-g6tw2\" (UID: \"5d310f0a-e80a-40e8-8092-dfa4518a077a\") " pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:30.239530 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.239509 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-webhook-cert\") pod \"maas-controller-d848cbdf4-g6tw2\" (UID: \"5d310f0a-e80a-40e8-8092-dfa4518a077a\") " pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:30.239645 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.239550 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-metrics-certs\") pod \"maas-controller-d848cbdf4-g6tw2\" (UID: \"5d310f0a-e80a-40e8-8092-dfa4518a077a\") " pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:30.299626 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.299596 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz8g8\" (UniqueName: \"kubernetes.io/projected/5d310f0a-e80a-40e8-8092-dfa4518a077a-kube-api-access-dz8g8\") pod \"maas-controller-d848cbdf4-g6tw2\" (UID: \"5d310f0a-e80a-40e8-8092-dfa4518a077a\") " pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:30.368681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.368661 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:30.743223 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.743090 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d848cbdf4-g6tw2"] Nov 03 07:49:30.745956 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:49:30.745914 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d310f0a_e80a_40e8_8092_dfa4518a077a.slice/crio-c52b2037d6dc2b60f97d94163141a1343b0f430d257552c688ac4fbbf355e973 WatchSource:0}: Error finding container c52b2037d6dc2b60f97d94163141a1343b0f430d257552c688ac4fbbf355e973: Status 404 returned error can't find the container with id c52b2037d6dc2b60f97d94163141a1343b0f430d257552c688ac4fbbf355e973 Nov 03 07:49:30.747497 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.747474 4409 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:49:30.900117 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.900084 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d848cbdf4-g6tw2" event={"ID":"5d310f0a-e80a-40e8-8092-dfa4518a077a","Type":"ContainerStarted","Data":"c52b2037d6dc2b60f97d94163141a1343b0f430d257552c688ac4fbbf355e973"} Nov 03 07:49:30.910835 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:30.910809 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6dbc02a-e589-4172-8071-272bd1a343d4" path="/var/lib/kubelet/pods/f6dbc02a-e589-4172-8071-272bd1a343d4/volumes" Nov 03 07:49:31.906578 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:31.906489 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d848cbdf4-g6tw2" event={"ID":"5d310f0a-e80a-40e8-8092-dfa4518a077a","Type":"ContainerStarted","Data":"de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840"} Nov 03 07:49:31.906578 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:31.906550 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:32.027823 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:32.027773 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-d848cbdf4-g6tw2" podStartSLOduration=1.201609701 podStartE2EDuration="2.027758226s" totalImagesPullingTime="826.148525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:49:32.025702725 +0000 UTC m=+3027.218244478" watchObservedRunningTime="2026-11-03 07:49:32.027758226 +0000 UTC m=+3027.220299976" Nov 03 07:49:38.774329 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:49:38.774283 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:49:42.914122 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:42.914092 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:49:44.271239 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.271205 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl"] Nov 03 07:49:44.271654 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.271608 4409 cpu_manager.go:414] "containerMap: removing container" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerName="istio-proxy" Nov 03 07:49:44.271654 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.271620 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerName="istio-proxy" Nov 03 07:49:44.271768 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.271693 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa1ee478-5863-4f95-8512-d6a2f5c4f6ed" containerName="istio-proxy" Nov 03 07:49:44.274675 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.274656 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.320301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.320270 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc"] Nov 03 07:49:44.323700 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.323678 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.365312 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.365270 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.365589 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.365564 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.365850 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.365823 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.365964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.365873 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.365964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.365907 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.365964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.365945 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.366098 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.366009 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpbhx\" (UniqueName: \"kubernetes.io/projected/599908f3-c725-49f4-9151-5e780af03db6-kube-api-access-zpbhx\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.366098 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.366050 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.366181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.366117 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.366181 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.366144 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599908f3-c725-49f4-9151-5e780af03db6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.366264 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.366250 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.366310 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.366271 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599908f3-c725-49f4-9151-5e780af03db6-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.366310 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.366295 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599908f3-c725-49f4-9151-5e780af03db6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.366387 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.366318 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.366429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.366384 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtd9\" (UniqueName: \"kubernetes.io/projected/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-kube-api-access-frtd9\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.366429 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.366410 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.366507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.366429 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.366507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.366451 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.415049 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.415017 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc"] Nov 03 07:49:44.467184 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467144 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.467184 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467184 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599908f3-c725-49f4-9151-5e780af03db6-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.467464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467207 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599908f3-c725-49f4-9151-5e780af03db6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.467464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467222 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.467464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467350 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtd9\" (UniqueName: \"kubernetes.io/projected/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-kube-api-access-frtd9\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.467464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467395 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.467464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467425 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.467464 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467448 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.467819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467490 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.467819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467520 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.467819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467568 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.467819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467679 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.467819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467700 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.467819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467767 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.467819 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467810 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.468160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467836 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.468160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467846 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.468160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467884 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbhx\" (UniqueName: \"kubernetes.io/projected/599908f3-c725-49f4-9151-5e780af03db6-kube-api-access-zpbhx\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.468160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467917 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.468160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467974 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.468160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.467998 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599908f3-c725-49f4-9151-5e780af03db6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.468160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.468057 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.468160 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.468103 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.468792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.468307 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.468792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.468429 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.468792 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.468649 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.468967 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.468811 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.468967 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.468900 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599908f3-c725-49f4-9151-5e780af03db6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.470278 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.470252 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.470514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.470495 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.470652 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.470623 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599908f3-c725-49f4-9151-5e780af03db6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.470815 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.470798 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599908f3-c725-49f4-9151-5e780af03db6-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.488642 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.488614 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl"] Nov 03 07:49:44.530898 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.530825 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.531049 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.530935 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbhx\" (UniqueName: \"kubernetes.io/projected/599908f3-c725-49f4-9151-5e780af03db6-kube-api-access-zpbhx\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.531049 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.530991 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599908f3-c725-49f4-9151-5e780af03db6-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tq7bl\" (UID: \"599908f3-c725-49f4-9151-5e780af03db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.531129 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.531114 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtd9\" (UniqueName: \"kubernetes.io/projected/56224d4f-9ba8-4f4d-a938-0e5dbc6c6504-kube-api-access-frtd9\") pod \"maas-default-gateway-openshift-default-58d898bd55-f66wc\" (UID: \"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:44.587750 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.587699 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:44.636055 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:44.636023 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:45.028860 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:45.028836 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl"] Nov 03 07:49:45.030478 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:49:45.030441 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599908f3_c725_49f4_9151_5e780af03db6.slice/crio-3028b2300a6ea538618da8fe0fe534f6d9fd0e8acb22e83abf0d41edfbcba0ce WatchSource:0}: Error finding container 3028b2300a6ea538618da8fe0fe534f6d9fd0e8acb22e83abf0d41edfbcba0ce: Status 404 returned error can't find the container with id 3028b2300a6ea538618da8fe0fe534f6d9fd0e8acb22e83abf0d41edfbcba0ce Nov 03 07:49:45.032627 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:45.032597 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:49:45.032692 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:45.032662 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:49:45.032789 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:45.032692 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:49:45.082897 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:45.082875 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc"] Nov 03 07:49:45.085827 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:49:45.085791 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56224d4f_9ba8_4f4d_a938_0e5dbc6c6504.slice/crio-569f6b09aaff7b3343070b68e7274e620e4a31817b52e6c5bb2a72decae64142 WatchSource:0}: Error finding container 569f6b09aaff7b3343070b68e7274e620e4a31817b52e6c5bb2a72decae64142: Status 404 returned error can't find the container with id 569f6b09aaff7b3343070b68e7274e620e4a31817b52e6c5bb2a72decae64142 Nov 03 07:49:45.089680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:45.089250 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:49:45.089680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:45.089325 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:49:45.089680 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:45.089363 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:49:46.012788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:46.011785 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" event={"ID":"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504","Type":"ContainerStarted","Data":"e65efb566e1660da875c95fa48c9ccac318901035af93d9265f2d02225f21f06"} Nov 03 07:49:46.012788 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:46.011855 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" event={"ID":"56224d4f-9ba8-4f4d-a938-0e5dbc6c6504","Type":"ContainerStarted","Data":"569f6b09aaff7b3343070b68e7274e620e4a31817b52e6c5bb2a72decae64142"} Nov 03 07:49:46.015908 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:46.015299 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" event={"ID":"599908f3-c725-49f4-9151-5e780af03db6","Type":"ContainerStarted","Data":"22812c4dcd5a6d690808ca7036d2c3a5de1b1643608b3e2457ec4103fe61f915"} Nov 03 07:49:46.015908 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:46.015358 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" event={"ID":"599908f3-c725-49f4-9151-5e780af03db6","Type":"ContainerStarted","Data":"3028b2300a6ea538618da8fe0fe534f6d9fd0e8acb22e83abf0d41edfbcba0ce"} Nov 03 07:49:46.258801 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:46.258743 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" podStartSLOduration=2.25872933 podStartE2EDuration="2.25872933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:49:46.208361428 +0000 UTC m=+3041.400903179" watchObservedRunningTime="2026-11-03 07:49:46.25872933 +0000 UTC m=+3041.451271079" Nov 03 07:49:46.333062 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:46.333001 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" podStartSLOduration=2.332987101 podStartE2EDuration="2.332987101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:49:46.331271633 +0000 UTC m=+3041.523813384" watchObservedRunningTime="2026-11-03 07:49:46.332987101 +0000 UTC m=+3041.525528850" Nov 03 07:49:46.592412 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:46.590553 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:46.592412 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:46.590683 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 07:49:46.592412 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:46.590750 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" podUID="599908f3-c725-49f4-9151-5e780af03db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 07:49:46.636229 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:46.636193 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:46.637418 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:46.637396 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f66wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 03 07:49:46.637475 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:46.637441 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" podUID="56224d4f-9ba8-4f4d-a938-0e5dbc6c6504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 03 07:49:47.592738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:47.592092 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 07:49:47.592738 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:47.592163 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" podUID="599908f3-c725-49f4-9151-5e780af03db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 07:49:47.636611 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:47.636578 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f66wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 03 07:49:47.636800 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:47.636622 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" podUID="56224d4f-9ba8-4f4d-a938-0e5dbc6c6504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 03 07:49:48.593262 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:48.593225 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 07:49:48.593894 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:48.593860 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" podUID="599908f3-c725-49f4-9151-5e780af03db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 07:49:48.637354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:48.637320 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f66wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 03 07:49:48.637514 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:48.637391 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" podUID="56224d4f-9ba8-4f4d-a938-0e5dbc6c6504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 03 07:49:49.604110 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:49.604074 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 07:49:49.604687 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:49.604657 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" podUID="599908f3-c725-49f4-9151-5e780af03db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 07:49:49.636875 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:49.636845 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f66wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 03 07:49:49.637042 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:49.636897 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" podUID="56224d4f-9ba8-4f4d-a938-0e5dbc6c6504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 03 07:49:50.592230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:50.592195 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 07:49:50.592555 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:50.592509 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" podUID="599908f3-c725-49f4-9151-5e780af03db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 07:49:50.636584 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:50.636554 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f66wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 03 07:49:50.637046 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:50.637025 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" podUID="56224d4f-9ba8-4f4d-a938-0e5dbc6c6504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 03 07:49:51.588857 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:51.588823 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 03 07:49:51.589054 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:51.588885 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" podUID="599908f3-c725-49f4-9151-5e780af03db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 03 07:49:51.637086 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:51.637047 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f66wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 03 07:49:51.637447 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:51.637115 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" podUID="56224d4f-9ba8-4f4d-a938-0e5dbc6c6504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 03 07:49:52.637288 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:52.637247 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f66wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 03 07:49:52.637811 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:52.637328 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" podUID="56224d4f-9ba8-4f4d-a938-0e5dbc6c6504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 03 07:49:53.340597 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:53.340565 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:53.341087 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:53.341038 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:53.341894 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:53.341873 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" Nov 03 07:49:53.641090 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:53.641014 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:53.641580 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:53.641380 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:49:53.642103 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:49:53.642086 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" Nov 03 07:50:03.739120 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:03.739088 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d848cbdf4-g6tw2"] Nov 03 07:50:03.739594 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:03.739279 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-d848cbdf4-g6tw2" podUID="5d310f0a-e80a-40e8-8092-dfa4518a077a" containerName="manager" containerID="cri-o://de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840" gracePeriod=30 Nov 03 07:50:03.992983 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:03.992908 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:50:04.091681 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.091651 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-webhook-cert\") pod \"5d310f0a-e80a-40e8-8092-dfa4518a077a\" (UID: \"5d310f0a-e80a-40e8-8092-dfa4518a077a\") " Nov 03 07:50:04.091864 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.091822 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d310f0a-e80a-40e8-8092-dfa4518a077a-kube-api-access-dz8g8\" (UniqueName: \"kubernetes.io/projected/5d310f0a-e80a-40e8-8092-dfa4518a077a-kube-api-access-dz8g8\") pod \"5d310f0a-e80a-40e8-8092-dfa4518a077a\" (UID: \"5d310f0a-e80a-40e8-8092-dfa4518a077a\") " Nov 03 07:50:04.091937 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.091887 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-metrics-certs\") pod \"5d310f0a-e80a-40e8-8092-dfa4518a077a\" (UID: \"5d310f0a-e80a-40e8-8092-dfa4518a077a\") " Nov 03 07:50:04.093975 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.093947 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-webhook-cert" pod "5d310f0a-e80a-40e8-8092-dfa4518a077a" (UID: "5d310f0a-e80a-40e8-8092-dfa4518a077a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:50:04.094080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.094039 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d310f0a-e80a-40e8-8092-dfa4518a077a-kube-api-access-dz8g8" pod "5d310f0a-e80a-40e8-8092-dfa4518a077a" (UID: "5d310f0a-e80a-40e8-8092-dfa4518a077a"). InnerVolumeSpecName "kube-api-access-dz8g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:50:04.094080 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.094051 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-metrics-certs" pod "5d310f0a-e80a-40e8-8092-dfa4518a077a" (UID: "5d310f0a-e80a-40e8-8092-dfa4518a077a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:50:04.110026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.110002 4409 generic.go:356] "Generic (PLEG): container finished" podID="5d310f0a-e80a-40e8-8092-dfa4518a077a" containerID="de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840" exitCode=0 Nov 03 07:50:04.110127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.110046 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d848cbdf4-g6tw2" Nov 03 07:50:04.110127 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.110079 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d848cbdf4-g6tw2" event={"ID":"5d310f0a-e80a-40e8-8092-dfa4518a077a","Type":"ContainerDied","Data":"de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840"} Nov 03 07:50:04.110206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.110132 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d848cbdf4-g6tw2" event={"ID":"5d310f0a-e80a-40e8-8092-dfa4518a077a","Type":"ContainerDied","Data":"c52b2037d6dc2b60f97d94163141a1343b0f430d257552c688ac4fbbf355e973"} Nov 03 07:50:04.110206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.110155 4409 scope.go:122] "RemoveContainer" containerID="de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840" Nov 03 07:50:04.120989 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.120967 4409 scope.go:122] "RemoveContainer" containerID="de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840" Nov 03 07:50:04.121389 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:50:04.121341 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840\": container with ID starting with de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840 not found: ID does not exist" containerID="de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840" Nov 03 07:50:04.121481 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.121398 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840"} err="failed to get container status \"de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840\": rpc error: code = NotFound desc = could not find container \"de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840\": container with ID starting with de89c027cacd3e612e4ed75377f356726a952b7ea6874ef9c73592bda4e95840 not found: ID does not exist" Nov 03 07:50:04.193348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.193319 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:50:04.193348 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.193351 4409 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d310f0a-e80a-40e8-8092-dfa4518a077a-webhook-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:50:04.193535 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.193366 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz8g8\" (UniqueName: \"kubernetes.io/projected/5d310f0a-e80a-40e8-8092-dfa4518a077a-kube-api-access-dz8g8\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:50:04.282234 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.282161 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d848cbdf4-g6tw2"] Nov 03 07:50:04.288276 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.288251 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d848cbdf4-g6tw2"] Nov 03 07:50:04.911765 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:04.911731 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d310f0a-e80a-40e8-8092-dfa4518a077a" path="/var/lib/kubelet/pods/5d310f0a-e80a-40e8-8092-dfa4518a077a/volumes" Nov 03 07:50:05.728760 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.728722 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d848cbdf4-6dpvx"] Nov 03 07:50:05.729175 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.729160 4409 cpu_manager.go:414] "containerMap: removing container" podUID="5d310f0a-e80a-40e8-8092-dfa4518a077a" containerName="manager" Nov 03 07:50:05.729238 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.729177 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d310f0a-e80a-40e8-8092-dfa4518a077a" containerName="manager" Nov 03 07:50:05.729273 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.729256 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d310f0a-e80a-40e8-8092-dfa4518a077a" containerName="manager" Nov 03 07:50:05.732354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.732334 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:05.835255 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.835216 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7pc74\"" Nov 03 07:50:05.835404 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.835256 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 07:50:05.835404 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.835330 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 07:50:05.835404 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.835391 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 07:50:05.838407 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.838386 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d848cbdf4-6dpvx"] Nov 03 07:50:05.910165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.910139 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a490802e-9166-438d-b140-60fe7ced2b9a-metrics-certs\") pod \"maas-controller-d848cbdf4-6dpvx\" (UID: \"a490802e-9166-438d-b140-60fe7ced2b9a\") " pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:05.910301 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.910191 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9b96\" (UniqueName: \"kubernetes.io/projected/a490802e-9166-438d-b140-60fe7ced2b9a-kube-api-access-j9b96\") pod \"maas-controller-d848cbdf4-6dpvx\" (UID: \"a490802e-9166-438d-b140-60fe7ced2b9a\") " pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:05.910344 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:05.910325 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a490802e-9166-438d-b140-60fe7ced2b9a-webhook-cert\") pod \"maas-controller-d848cbdf4-6dpvx\" (UID: \"a490802e-9166-438d-b140-60fe7ced2b9a\") " pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:06.011269 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:06.011187 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a490802e-9166-438d-b140-60fe7ced2b9a-webhook-cert\") pod \"maas-controller-d848cbdf4-6dpvx\" (UID: \"a490802e-9166-438d-b140-60fe7ced2b9a\") " pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:06.011269 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:06.011236 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a490802e-9166-438d-b140-60fe7ced2b9a-metrics-certs\") pod \"maas-controller-d848cbdf4-6dpvx\" (UID: \"a490802e-9166-438d-b140-60fe7ced2b9a\") " pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:06.011827 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:06.011280 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9b96\" (UniqueName: \"kubernetes.io/projected/a490802e-9166-438d-b140-60fe7ced2b9a-kube-api-access-j9b96\") pod \"maas-controller-d848cbdf4-6dpvx\" (UID: \"a490802e-9166-438d-b140-60fe7ced2b9a\") " pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:06.013898 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:06.013872 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a490802e-9166-438d-b140-60fe7ced2b9a-webhook-cert\") pod \"maas-controller-d848cbdf4-6dpvx\" (UID: \"a490802e-9166-438d-b140-60fe7ced2b9a\") " pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:06.013992 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:06.013967 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a490802e-9166-438d-b140-60fe7ced2b9a-metrics-certs\") pod \"maas-controller-d848cbdf4-6dpvx\" (UID: \"a490802e-9166-438d-b140-60fe7ced2b9a\") " pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:06.074442 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:06.074405 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9b96\" (UniqueName: \"kubernetes.io/projected/a490802e-9166-438d-b140-60fe7ced2b9a-kube-api-access-j9b96\") pod \"maas-controller-d848cbdf4-6dpvx\" (UID: \"a490802e-9166-438d-b140-60fe7ced2b9a\") " pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:06.344995 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:06.344965 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:06.770859 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:06.770835 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d848cbdf4-6dpvx"] Nov 03 07:50:06.773311 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:50:06.773288 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda490802e_9166_438d_b140_60fe7ced2b9a.slice/crio-b2340885e8ae85c88c5f4eb650d861a25a22dfe9f402fd202fea611f5fd7f9b2 WatchSource:0}: Error finding container b2340885e8ae85c88c5f4eb650d861a25a22dfe9f402fd202fea611f5fd7f9b2: Status 404 returned error can't find the container with id b2340885e8ae85c88c5f4eb650d861a25a22dfe9f402fd202fea611f5fd7f9b2 Nov 03 07:50:07.126805 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:07.126774 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d848cbdf4-6dpvx" event={"ID":"a490802e-9166-438d-b140-60fe7ced2b9a","Type":"ContainerStarted","Data":"b2340885e8ae85c88c5f4eb650d861a25a22dfe9f402fd202fea611f5fd7f9b2"} Nov 03 07:50:08.133982 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:08.133946 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d848cbdf4-6dpvx" event={"ID":"a490802e-9166-438d-b140-60fe7ced2b9a","Type":"ContainerStarted","Data":"0246c81feb9c6656bb8932e6e26042f822fc5e55093fadea1101f2adcd2e57d2"} Nov 03 07:50:08.134437 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:08.134078 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:08.255388 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:08.255338 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-d848cbdf4-6dpvx" podStartSLOduration=2.441579718 podStartE2EDuration="3.255323937s" totalImagesPullingTime="813.744219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:50:08.253323832 +0000 UTC m=+3063.445865564" watchObservedRunningTime="2026-11-03 07:50:08.255323937 +0000 UTC m=+3063.447865686" Nov 03 07:50:08.774297 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:50:08.774259 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:50:19.141084 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:50:19.141054 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d848cbdf4-6dpvx" Nov 03 07:50:38.774852 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:50:38.774811 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:51:08.773103 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:51:08.773067 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:51:38.774509 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:51:38.774460 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:52:08.777085 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:52:08.777048 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:52:38.775210 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:52:38.775169 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:53:08.772878 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:53:08.772828 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:53:38.774435 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:53:38.774391 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:54:05.415766 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:54:05.415736 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:54:05.418257 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:54:05.418230 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:54:08.773567 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:54:08.773521 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:54:38.775149 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:54:38.775108 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:54:54.386739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:54:54.385842 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tq7bl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 07:54:54.386739 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:54:54.385953 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tq7bl" podUID="599908f3-c725-49f4-9151-5e780af03db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 07:55:08.773918 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:55:08.773878 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:55:38.773152 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:55:38.773110 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:56:08.773923 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:56:08.773886 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:56:38.773866 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:56:38.773826 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:57:06.400164 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.400133 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz"] Nov 03 07:57:06.403851 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.403831 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.464153 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.464120 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-98d4f3-openshift-default-dockercfg-sgnk6\"" Nov 03 07:57:06.494027 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.493992 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-podinfo\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.494186 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.494035 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1882307-36b0-425b-8b61-7cc2217afe6c-istiod-ca-cert\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.494186 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.494102 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-data\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.494186 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.494135 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-token\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.494186 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.494165 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qwvn\" (UniqueName: \"kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-kube-api-access-9qwvn\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.494392 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.494358 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-credential-socket\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.494480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.494462 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-socket\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.494552 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.494535 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-envoy\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.494602 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.494585 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-certs\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.516061 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.516036 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz"] Nov 03 07:57:06.595117 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595081 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1882307-36b0-425b-8b61-7cc2217afe6c-istiod-ca-cert\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.595117 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595119 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-data\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.595354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595150 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-token\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.595354 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595188 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qwvn\" (UniqueName: \"kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-kube-api-access-9qwvn\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.595461 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595410 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-credential-socket\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.595461 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595449 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-data\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.595575 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595507 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-socket\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.595634 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595583 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-envoy\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.595693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595632 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-certs\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.595693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595672 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-credential-socket\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.595693 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595684 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-podinfo\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.595905 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595882 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1882307-36b0-425b-8b61-7cc2217afe6c-istiod-ca-cert\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.595968 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595882 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-socket\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.596026 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.595997 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-certs\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.598097 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.598078 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-envoy\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.598338 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.598320 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-podinfo\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.657797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.657740 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-token\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.657797 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.657756 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qwvn\" (UniqueName: \"kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-kube-api-access-9qwvn\") pod \"e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:06.723013 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:06.722505 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:07.154427 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:07.154403 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz"] Nov 03 07:57:07.156198 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:57:07.156174 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1882307_36b0_425b_8b61_7cc2217afe6c.slice/crio-b84fab87620e1a8b1c08ff013d7eb0703553aa562677f8426875986c5567d63f WatchSource:0}: Error finding container b84fab87620e1a8b1c08ff013d7eb0703553aa562677f8426875986c5567d63f: Status 404 returned error can't find the container with id b84fab87620e1a8b1c08ff013d7eb0703553aa562677f8426875986c5567d63f Nov 03 07:57:07.158431 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:07.158365 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:57:07.158530 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:07.158471 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:57:07.158598 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:07.158546 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:57:07.319566 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:07.319532 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" event={"ID":"f1882307-36b0-425b-8b61-7cc2217afe6c","Type":"ContainerStarted","Data":"e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e"} Nov 03 07:57:07.319756 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:07.319589 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" event={"ID":"f1882307-36b0-425b-8b61-7cc2217afe6c","Type":"ContainerStarted","Data":"b84fab87620e1a8b1c08ff013d7eb0703553aa562677f8426875986c5567d63f"} Nov 03 07:57:07.443641 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:07.443534 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" podStartSLOduration=1.443515267 podStartE2EDuration="1.443515267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:57:07.442639815 +0000 UTC m=+3482.635181566" watchObservedRunningTime="2026-11-03 07:57:07.443515267 +0000 UTC m=+3482.636057018" Nov 03 07:57:07.723648 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:07.723565 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:08.728367 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:08.728338 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:08.772236 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:57:08.772201 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:57:09.329536 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:09.329505 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:09.330480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:09.330458 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:16.753703 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.753657 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447"] Nov 03 07:57:16.757456 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.757434 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:16.812648 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.812625 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-98d4f3-serving-cert\"" Nov 03 07:57:16.863246 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.863216 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447"] Nov 03 07:57:16.894604 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.894573 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a402e0d1-f05b-48e1-8259-6ec1e9f51208-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:16.894759 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.894646 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:16.894759 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.894694 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-metrics-certs\") pod \"maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:16.894877 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.894857 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjxww\" (UniqueName: \"kubernetes.io/projected/a402e0d1-f05b-48e1-8259-6ec1e9f51208-kube-api-access-fjxww\") pod \"maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:16.995806 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.995773 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a402e0d1-f05b-48e1-8259-6ec1e9f51208-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:16.995981 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.995822 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:16.995981 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.995837 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-metrics-certs\") pod \"maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:16.995981 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.995868 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjxww\" (UniqueName: \"kubernetes.io/projected/a402e0d1-f05b-48e1-8259-6ec1e9f51208-kube-api-access-fjxww\") pod \"maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:16.996492 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.996467 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a402e0d1-f05b-48e1-8259-6ec1e9f51208-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:16.998420 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.998401 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-metrics-certs\") pod \"maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:16.998572 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:16.998553 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:17.059432 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:17.059403 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjxww\" (UniqueName: \"kubernetes.io/projected/a402e0d1-f05b-48e1-8259-6ec1e9f51208-kube-api-access-fjxww\") pod \"maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:17.071009 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:17.070987 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:17.449224 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:17.449200 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447"] Nov 03 07:57:17.451686 ip-10-0-138-121 kubenswrapper[4409]: W1103 07:57:17.451643 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda402e0d1_f05b_48e1_8259_6ec1e9f51208.slice/crio-a1e809187dd4daa0bc9fcd25b5e2eb3754ba1fa552bb025525271f3f89875ff7 WatchSource:0}: Error finding container a1e809187dd4daa0bc9fcd25b5e2eb3754ba1fa552bb025525271f3f89875ff7: Status 404 returned error can't find the container with id a1e809187dd4daa0bc9fcd25b5e2eb3754ba1fa552bb025525271f3f89875ff7 Nov 03 07:57:17.453090 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:17.453071 4409 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 07:57:18.376350 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:18.376321 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" event={"ID":"a402e0d1-f05b-48e1-8259-6ec1e9f51208","Type":"ContainerStarted","Data":"a1e809187dd4daa0bc9fcd25b5e2eb3754ba1fa552bb025525271f3f89875ff7"} Nov 03 07:57:19.382505 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:19.382469 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" event={"ID":"a402e0d1-f05b-48e1-8259-6ec1e9f51208","Type":"ContainerStarted","Data":"b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4"} Nov 03 07:57:19.382932 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:19.382576 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:19.502455 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:19.502405 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" podStartSLOduration=2.63815487 podStartE2EDuration="3.502391635s" totalImagesPullingTime="864.236765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:57:19.500565728 +0000 UTC m=+3494.693107478" watchObservedRunningTime="2026-11-03 07:57:19.502391635 +0000 UTC m=+3494.694933384" Nov 03 07:57:25.393696 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:25.393665 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:57:35.458620 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:35.458589 4409 generic.go:356] "Generic (PLEG): container finished" podID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerID="e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e" exitCode=0 Nov 03 07:57:35.459146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:35.458671 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" event={"ID":"f1882307-36b0-425b-8b61-7cc2217afe6c","Type":"ContainerDied","Data":"e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e"} Nov 03 07:57:35.459146 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:35.459139 4409 scope.go:122] "RemoveContainer" containerID="e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e" Nov 03 07:57:35.460496 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:35.460460 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:57:35.460608 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:35.460528 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:57:35.460608 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:35.460559 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 07:57:36.480358 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:36.480311 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" event={"ID":"f1882307-36b0-425b-8b61-7cc2217afe6c","Type":"ContainerStarted","Data":"e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d"} Nov 03 07:57:36.723894 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:36.723855 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:36.725083 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:36.725054 4409 patch_prober.go:29] interesting pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 07:57:36.725229 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:36.725116 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 07:57:37.723891 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:37.723848 4409 patch_prober.go:29] interesting pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 07:57:37.724413 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:37.723912 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 07:57:38.724007 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:38.723972 4409 patch_prober.go:29] interesting pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 07:57:38.724449 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:38.724026 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 07:57:38.795769 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:57:38.795699 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:57:39.330552 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:39.330508 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:39.724613 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:39.724523 4409 patch_prober.go:29] interesting pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 07:57:39.724613 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:39.724581 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 07:57:40.724270 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:40.724228 4409 patch_prober.go:29] interesting pod/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 07:57:40.724468 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:40.724289 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 07:57:41.728258 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:41.728225 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:57:41.729202 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:57:41.729183 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:58:08.774007 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:58:08.773965 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:58:38.777132 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:58:38.777088 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:58:49.852599 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:49.852565 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447"] Nov 03 07:58:49.853135 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:49.852805 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" podUID="a402e0d1-f05b-48e1-8259-6ec1e9f51208" containerName="maas-api" containerID="cri-o://b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4" gracePeriod=30 Nov 03 07:58:50.388393 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:50.388354 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" podUID="a402e0d1-f05b-48e1-8259-6ec1e9f51208" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.164:8443/health\": dial tcp 10.133.0.164:8443: connect: connection refused" Nov 03 07:58:51.095206 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.095183 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:58:51.245614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.245544 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a402e0d1-f05b-48e1-8259-6ec1e9f51208-kube-api-access-fjxww\" (UniqueName: \"kubernetes.io/projected/a402e0d1-f05b-48e1-8259-6ec1e9f51208-kube-api-access-fjxww\") pod \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " Nov 03 07:58:51.245614 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.245581 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-maas-api-tls\") pod \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " Nov 03 07:58:51.245866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.245636 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-metrics-certs\") pod \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " Nov 03 07:58:51.245866 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.245668 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a402e0d1-f05b-48e1-8259-6ec1e9f51208-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a402e0d1-f05b-48e1-8259-6ec1e9f51208-openshift-service-ca\") pod \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\" (UID: \"a402e0d1-f05b-48e1-8259-6ec1e9f51208\") " Nov 03 07:58:51.246165 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.246128 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a402e0d1-f05b-48e1-8259-6ec1e9f51208-openshift-service-ca" pod "a402e0d1-f05b-48e1-8259-6ec1e9f51208" (UID: "a402e0d1-f05b-48e1-8259-6ec1e9f51208"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:58:51.247915 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.247889 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-maas-api-tls" pod "a402e0d1-f05b-48e1-8259-6ec1e9f51208" (UID: "a402e0d1-f05b-48e1-8259-6ec1e9f51208"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:58:51.248011 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.247924 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a402e0d1-f05b-48e1-8259-6ec1e9f51208-kube-api-access-fjxww" pod "a402e0d1-f05b-48e1-8259-6ec1e9f51208" (UID: "a402e0d1-f05b-48e1-8259-6ec1e9f51208"). InnerVolumeSpecName "kube-api-access-fjxww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:58:51.248011 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.247962 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-metrics-certs" pod "a402e0d1-f05b-48e1-8259-6ec1e9f51208" (UID: "a402e0d1-f05b-48e1-8259-6ec1e9f51208"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 07:58:51.346960 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.346930 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjxww\" (UniqueName: \"kubernetes.io/projected/a402e0d1-f05b-48e1-8259-6ec1e9f51208-kube-api-access-fjxww\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:58:51.346960 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.346956 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:58:51.346960 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.346967 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a402e0d1-f05b-48e1-8259-6ec1e9f51208-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:58:51.347172 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.346975 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a402e0d1-f05b-48e1-8259-6ec1e9f51208-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:58:51.854635 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.854600 4409 generic.go:356] "Generic (PLEG): container finished" podID="a402e0d1-f05b-48e1-8259-6ec1e9f51208" containerID="b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4" exitCode=0 Nov 03 07:58:51.854865 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.854649 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" Nov 03 07:58:51.854865 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.854656 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" event={"ID":"a402e0d1-f05b-48e1-8259-6ec1e9f51208","Type":"ContainerDied","Data":"b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4"} Nov 03 07:58:51.854865 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.854706 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447" event={"ID":"a402e0d1-f05b-48e1-8259-6ec1e9f51208","Type":"ContainerDied","Data":"a1e809187dd4daa0bc9fcd25b5e2eb3754ba1fa552bb025525271f3f89875ff7"} Nov 03 07:58:51.854865 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.854757 4409 scope.go:122] "RemoveContainer" containerID="b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4" Nov 03 07:58:51.864653 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.864636 4409 scope.go:122] "RemoveContainer" containerID="b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4" Nov 03 07:58:51.864965 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:58:51.864946 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4\": container with ID starting with b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4 not found: ID does not exist" containerID="b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4" Nov 03 07:58:51.865044 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:51.864976 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4"} err="failed to get container status \"b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4\": rpc error: code = NotFound desc = could not find container \"b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4\": container with ID starting with b151e67f90788fad67348c67280b18dc267e8d1eb93710578d5af8886b5771a4 not found: ID does not exist" Nov 03 07:58:52.027480 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:52.027440 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447"] Nov 03 07:58:52.077063 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:52.077033 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-98d4f3-8578b89676-mw447"] Nov 03 07:58:52.911230 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:52.911198 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a402e0d1-f05b-48e1-8259-6ec1e9f51208" path="/var/lib/kubelet/pods/a402e0d1-f05b-48e1-8259-6ec1e9f51208/volumes" Nov 03 07:58:59.646390 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:59.646352 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz"] Nov 03 07:58:59.646983 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:58:59.646613 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerName="istio-proxy" containerID="cri-o://e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d" gracePeriod=30 Nov 03 07:59:04.896424 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.896395 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:59:04.929362 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.929295 4409 generic.go:356] "Generic (PLEG): container finished" podID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerID="e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d" exitCode=0 Nov 03 07:59:04.929362 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.929344 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" Nov 03 07:59:04.929507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.929343 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" event={"ID":"f1882307-36b0-425b-8b61-7cc2217afe6c","Type":"ContainerDied","Data":"e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d"} Nov 03 07:59:04.929507 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.929483 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz" event={"ID":"f1882307-36b0-425b-8b61-7cc2217afe6c","Type":"ContainerDied","Data":"b84fab87620e1a8b1c08ff013d7eb0703553aa562677f8426875986c5567d63f"} Nov 03 07:59:04.929575 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.929516 4409 scope.go:122] "RemoveContainer" containerID="e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d" Nov 03 07:59:04.938203 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.938187 4409 scope.go:122] "RemoveContainer" containerID="e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e" Nov 03 07:59:04.946250 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.946230 4409 scope.go:122] "RemoveContainer" containerID="e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d" Nov 03 07:59:04.946505 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:59:04.946480 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d\": container with ID starting with e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d not found: ID does not exist" containerID="e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d" Nov 03 07:59:04.946560 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.946506 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d"} err="failed to get container status \"e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d\": rpc error: code = NotFound desc = could not find container \"e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d\": container with ID starting with e706c62be2c2e627cdacc46f0356dfbe846dd0f0abc022b4900ff73373549a6d not found: ID does not exist" Nov 03 07:59:04.946560 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.946523 4409 scope.go:122] "RemoveContainer" containerID="e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e" Nov 03 07:59:04.946802 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:59:04.946786 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e\": container with ID starting with e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e not found: ID does not exist" containerID="e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e" Nov 03 07:59:04.946857 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.946807 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e"} err="failed to get container status \"e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e\": rpc error: code = NotFound desc = could not find container \"e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e\": container with ID starting with e2d442c0535b7f94ed833f4331b0d018e1d1e37c694b6255ec699680c07d215e not found: ID does not exist" Nov 03 07:59:04.974109 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.974089 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-token\" (UniqueName: \"kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-token\") pod \"f1882307-36b0-425b-8b61-7cc2217afe6c\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " Nov 03 07:59:04.974218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.974124 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-envoy\") pod \"f1882307-36b0-425b-8b61-7cc2217afe6c\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " Nov 03 07:59:04.974218 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.974179 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-podinfo\") pod \"f1882307-36b0-425b-8b61-7cc2217afe6c\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " Nov 03 07:59:04.974373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.974224 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1882307-36b0-425b-8b61-7cc2217afe6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1882307-36b0-425b-8b61-7cc2217afe6c-istiod-ca-cert\") pod \"f1882307-36b0-425b-8b61-7cc2217afe6c\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " Nov 03 07:59:04.974373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.974261 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-data\") pod \"f1882307-36b0-425b-8b61-7cc2217afe6c\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " Nov 03 07:59:04.974373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.974312 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-credential-socket\") pod \"f1882307-36b0-425b-8b61-7cc2217afe6c\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " Nov 03 07:59:04.974373 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.974353 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-certs\") pod \"f1882307-36b0-425b-8b61-7cc2217afe6c\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " Nov 03 07:59:04.974594 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.974378 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-kube-api-access-9qwvn\" (UniqueName: \"kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-kube-api-access-9qwvn\") pod \"f1882307-36b0-425b-8b61-7cc2217afe6c\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " Nov 03 07:59:04.974594 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.974417 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-socket\") pod \"f1882307-36b0-425b-8b61-7cc2217afe6c\" (UID: \"f1882307-36b0-425b-8b61-7cc2217afe6c\") " Nov 03 07:59:04.974919 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.974874 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-socket" pod "f1882307-36b0-425b-8b61-7cc2217afe6c" (UID: "f1882307-36b0-425b-8b61-7cc2217afe6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:59:04.974919 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.974874 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1882307-36b0-425b-8b61-7cc2217afe6c-istiod-ca-cert" pod "f1882307-36b0-425b-8b61-7cc2217afe6c" (UID: "f1882307-36b0-425b-8b61-7cc2217afe6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 07:59:04.975085 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.974953 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-certs" pod "f1882307-36b0-425b-8b61-7cc2217afe6c" (UID: "f1882307-36b0-425b-8b61-7cc2217afe6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:59:04.975085 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.975012 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-credential-socket" pod "f1882307-36b0-425b-8b61-7cc2217afe6c" (UID: "f1882307-36b0-425b-8b61-7cc2217afe6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:59:04.975085 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.975063 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-data" pod "f1882307-36b0-425b-8b61-7cc2217afe6c" (UID: "f1882307-36b0-425b-8b61-7cc2217afe6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:59:04.976405 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.976377 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-token" pod "f1882307-36b0-425b-8b61-7cc2217afe6c" (UID: "f1882307-36b0-425b-8b61-7cc2217afe6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:59:04.976964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.976943 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-envoy" pod "f1882307-36b0-425b-8b61-7cc2217afe6c" (UID: "f1882307-36b0-425b-8b61-7cc2217afe6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 07:59:04.976964 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.976949 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-podinfo" pod "f1882307-36b0-425b-8b61-7cc2217afe6c" (UID: "f1882307-36b0-425b-8b61-7cc2217afe6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 07:59:04.977126 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:04.977047 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-kube-api-access-9qwvn" pod "f1882307-36b0-425b-8b61-7cc2217afe6c" (UID: "f1882307-36b0-425b-8b61-7cc2217afe6c"). InnerVolumeSpecName "kube-api-access-9qwvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 07:59:05.075281 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.075254 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:59:05.075281 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.075277 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:59:05.075421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.075286 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:59:05.075421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.075297 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1882307-36b0-425b-8b61-7cc2217afe6c-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:59:05.075421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.075305 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:59:05.075421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.075314 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:59:05.075421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.075321 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:59:05.075421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.075330 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qwvn\" (UniqueName: \"kubernetes.io/projected/f1882307-36b0-425b-8b61-7cc2217afe6c-kube-api-access-9qwvn\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:59:05.075421 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.075339 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1882307-36b0-425b-8b61-7cc2217afe6c-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 07:59:05.353133 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.353101 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz"] Nov 03 07:59:05.359128 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.359105 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-98d4f3-openshift-default-648c896b4frb5rz"] Nov 03 07:59:05.448752 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.448727 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:59:05.451801 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:05.451780 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 07:59:06.910835 ip-10-0-138-121 kubenswrapper[4409]: I1103 07:59:06.910803 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" path="/var/lib/kubelet/pods/f1882307-36b0-425b-8b61-7cc2217afe6c/volumes" Nov 03 07:59:08.774067 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:59:08.774029 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 07:59:38.774984 ip-10-0-138-121 kubenswrapper[4409]: E1103 07:59:38.774943 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:00:08.772661 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:00:08.772619 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:00:38.774423 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:00:38.774383 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:01:08.773376 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:01:08.773340 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:01:24.691138 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:24.689801 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f66wc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:01:24.691923 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:24.691878 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" podUID="56224d4f-9ba8-4f4d-a938-0e5dbc6c6504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:01:38.774448 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:01:38.774404 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:01:46.676657 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.676619 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd"] Nov 03 08:01:46.677259 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.677241 4409 cpu_manager.go:414] "containerMap: removing container" podUID="a402e0d1-f05b-48e1-8259-6ec1e9f51208" containerName="maas-api" Nov 03 08:01:46.677339 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.677261 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a402e0d1-f05b-48e1-8259-6ec1e9f51208" containerName="maas-api" Nov 03 08:01:46.677339 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.677314 4409 cpu_manager.go:414] "containerMap: removing container" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerName="istio-proxy" Nov 03 08:01:46.677339 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.677321 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerName="istio-proxy" Nov 03 08:01:46.677339 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.677335 4409 cpu_manager.go:414] "containerMap: removing container" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerName="istio-proxy" Nov 03 08:01:46.677517 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.677343 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerName="istio-proxy" Nov 03 08:01:46.677517 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.677432 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerName="istio-proxy" Nov 03 08:01:46.677517 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.677443 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1882307-36b0-425b-8b61-7cc2217afe6c" containerName="istio-proxy" Nov 03 08:01:46.677517 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.677456 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="a402e0d1-f05b-48e1-8259-6ec1e9f51208" containerName="maas-api" Nov 03 08:01:46.680886 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.680868 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.736404 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.736371 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b435d8-openshift-default-dockercfg-fq242\"" Nov 03 08:01:46.744205 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.744170 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istiod-ca-cert\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.744205 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.744207 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-envoy\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.744445 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.744321 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rx8n\" (UniqueName: \"kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-kube-api-access-4rx8n\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.744445 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.744377 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-socket\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.744526 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.744453 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-podinfo\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.744526 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.744490 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-certs\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.744526 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.744505 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-credential-socket\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.744628 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.744527 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-token\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.744628 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.744547 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-data\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.793418 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.793386 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd"] Nov 03 08:01:46.845781 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.845703 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-podinfo\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.845966 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.845831 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-certs\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.845966 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.845859 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-credential-socket\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.845966 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.845900 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-token\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.845966 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.845932 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-data\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.846196 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.845994 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istiod-ca-cert\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.846196 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.846118 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-envoy\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.846371 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.846242 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rx8n\" (UniqueName: \"kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-kube-api-access-4rx8n\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.846371 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.846286 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-certs\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.846371 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.846290 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-socket\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.846552 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.846522 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-credential-socket\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.846632 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.846580 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-data\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.846832 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.846811 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-socket\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.846912 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.846826 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istiod-ca-cert\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.848422 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.848396 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-podinfo\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.848522 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.848480 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-envoy\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.909462 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.909429 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-token\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:46.909606 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:46.909509 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rx8n\" (UniqueName: \"kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-kube-api-access-4rx8n\") pod \"e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:47.010345 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:47.010261 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:47.459628 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:47.459507 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd"] Nov 03 08:01:47.462057 ip-10-0-138-121 kubenswrapper[4409]: W1103 08:01:47.462029 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f5a50b_b9c6_4b55_a11a_5b45749b1ba4.slice/crio-b1dc0d21d4f13d4fe31e10e6e7971fef3080c72a6ed75527cb1de3a8dfa99c80 WatchSource:0}: Error finding container b1dc0d21d4f13d4fe31e10e6e7971fef3080c72a6ed75527cb1de3a8dfa99c80: Status 404 returned error can't find the container with id b1dc0d21d4f13d4fe31e10e6e7971fef3080c72a6ed75527cb1de3a8dfa99c80 Nov 03 08:01:47.464143 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:47.464114 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:01:47.464240 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:47.464176 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:01:47.464240 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:47.464205 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:01:47.758938 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:47.758853 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" event={"ID":"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4","Type":"ContainerStarted","Data":"30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca"} Nov 03 08:01:47.758938 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:47.758914 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" event={"ID":"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4","Type":"ContainerStarted","Data":"b1dc0d21d4f13d4fe31e10e6e7971fef3080c72a6ed75527cb1de3a8dfa99c80"} Nov 03 08:01:47.884160 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:47.884112 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" podStartSLOduration=1.884098118 podStartE2EDuration="1.884098118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:47.880913448 +0000 UTC m=+3763.073455197" watchObservedRunningTime="2026-11-03 08:01:47.884098118 +0000 UTC m=+3763.076639935" Nov 03 08:01:48.023279 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:48.019771 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:48.024913 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:48.024885 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:48.764093 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:48.764066 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:48.764994 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:48.764977 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:01:57.143804 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.143771 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6"] Nov 03 08:01:57.147287 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.147269 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.201665 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.201638 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b435d8-serving-cert\"" Nov 03 08:01:57.247165 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.247135 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.247363 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.247202 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7fa7fc-5c26-444b-acbc-d288900869bc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.247363 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.247288 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.247363 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.247352 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x79p\" (UniqueName: \"kubernetes.io/projected/9b7fa7fc-5c26-444b-acbc-d288900869bc-kube-api-access-5x79p\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.252695 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.252672 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6"] Nov 03 08:01:57.348146 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.348114 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x79p\" (UniqueName: \"kubernetes.io/projected/9b7fa7fc-5c26-444b-acbc-d288900869bc-kube-api-access-5x79p\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.348357 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.348187 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.348357 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.348241 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7fa7fc-5c26-444b-acbc-d288900869bc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.348357 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.348271 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.348481 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:01:57.348369 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-serving-cert: secret "maas-api-e2e-worker-single-main-b435d8-serving-cert" not found Nov 03 08:01:57.348481 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:01:57.348447 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-maas-api-tls podName:9b7fa7fc-5c26-444b-acbc-d288900869bc nodeName:}" failed. No retries permitted until 2026-11-03 08:01:57.848424643 +0000 UTC m=+3773.040966384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-maas-api-tls") pod "maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" (UID: "9b7fa7fc-5c26-444b-acbc-d288900869bc") : secret "maas-api-e2e-worker-single-main-b435d8-serving-cert" not found Nov 03 08:01:57.348931 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.348896 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7fa7fc-5c26-444b-acbc-d288900869bc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.350952 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.350934 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.411640 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.411571 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x79p\" (UniqueName: \"kubernetes.io/projected/9b7fa7fc-5c26-444b-acbc-d288900869bc-kube-api-access-5x79p\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.858366 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.858335 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:57.860872 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:57.860845 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:58.060826 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:58.060789 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:58.439119 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:58.439095 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6"] Nov 03 08:01:58.441627 ip-10-0-138-121 kubenswrapper[4409]: W1103 08:01:58.441596 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7fa7fc_5c26_444b_acbc_d288900869bc.slice/crio-0f19f3849daf3b9e5e9184c77ab7836426e1c0ed67a921e0b1be0e882a97ee2a WatchSource:0}: Error finding container 0f19f3849daf3b9e5e9184c77ab7836426e1c0ed67a921e0b1be0e882a97ee2a: Status 404 returned error can't find the container with id 0f19f3849daf3b9e5e9184c77ab7836426e1c0ed67a921e0b1be0e882a97ee2a Nov 03 08:01:58.811581 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:58.811548 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" event={"ID":"9b7fa7fc-5c26-444b-acbc-d288900869bc","Type":"ContainerStarted","Data":"0f19f3849daf3b9e5e9184c77ab7836426e1c0ed67a921e0b1be0e882a97ee2a"} Nov 03 08:01:59.818809 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:59.818772 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" event={"ID":"9b7fa7fc-5c26-444b-acbc-d288900869bc","Type":"ContainerStarted","Data":"f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733"} Nov 03 08:01:59.819190 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:59.818881 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:01:59.940257 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:01:59.940208 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" podStartSLOduration=2.076741097 podStartE2EDuration="2.940192774s" totalImagesPullingTime="863.451677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:59.937933245 +0000 UTC m=+3775.130475004" watchObservedRunningTime="2026-11-03 08:01:59.940192774 +0000 UTC m=+3775.132734523" Nov 03 08:02:05.829376 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:05.829347 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:02:08.797122 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:02:08.797081 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:02:14.911956 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:14.911926 4409 generic.go:356] "Generic (PLEG): container finished" podID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerID="30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca" exitCode=0 Nov 03 08:02:14.912390 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:14.911928 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" event={"ID":"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4","Type":"ContainerDied","Data":"30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca"} Nov 03 08:02:14.912390 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:14.912279 4409 scope.go:122] "RemoveContainer" containerID="30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca" Nov 03 08:02:14.913292 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:14.913268 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:14.913343 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:14.913332 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:14.913378 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:14.913363 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:02:15.919265 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:15.919234 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" event={"ID":"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4","Type":"ContainerStarted","Data":"253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1"} Nov 03 08:02:16.011247 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:16.011204 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:02:16.012463 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:16.012430 4409 patch_prober.go:29] interesting pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 08:02:16.012600 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:16.012482 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 08:02:17.011525 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:17.011485 4409 patch_prober.go:29] interesting pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 08:02:17.012054 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:17.011537 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 08:02:18.011426 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:18.011386 4409 patch_prober.go:29] interesting pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 08:02:18.011615 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:18.011451 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 08:02:18.764664 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:18.764631 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:02:19.010794 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:19.010761 4409 patch_prober.go:29] interesting pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 08:02:19.010962 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:19.010815 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 08:02:20.011434 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:20.011397 4409 patch_prober.go:29] interesting pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 08:02:20.011880 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:20.011449 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 08:02:21.014936 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:21.014905 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:02:21.015918 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:02:21.015898 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:02:38.775372 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:02:38.775331 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:03:08.774931 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:03:08.774885 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:03:30.317324 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:30.317288 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6"] Nov 03 08:03:30.317892 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:30.317486 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" podUID="9b7fa7fc-5c26-444b-acbc-d288900869bc" containerName="maas-api" containerID="cri-o://f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733" gracePeriod=30 Nov 03 08:03:30.825778 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:30.825635 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" podUID="9b7fa7fc-5c26-444b-acbc-d288900869bc" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.166:8443/health\": dial tcp 10.133.0.166:8443: connect: connection refused" Nov 03 08:03:31.572450 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.572427 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:03:31.628962 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.628895 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7fa7fc-5c26-444b-acbc-d288900869bc-kube-api-access-5x79p\" (UniqueName: \"kubernetes.io/projected/9b7fa7fc-5c26-444b-acbc-d288900869bc-kube-api-access-5x79p\") pod \"9b7fa7fc-5c26-444b-acbc-d288900869bc\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " Nov 03 08:03:31.628962 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.628946 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-metrics-certs\") pod \"9b7fa7fc-5c26-444b-acbc-d288900869bc\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " Nov 03 08:03:31.629141 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.628983 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b7fa7fc-5c26-444b-acbc-d288900869bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7fa7fc-5c26-444b-acbc-d288900869bc-openshift-service-ca\") pod \"9b7fa7fc-5c26-444b-acbc-d288900869bc\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " Nov 03 08:03:31.629141 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.629030 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-maas-api-tls\") pod \"9b7fa7fc-5c26-444b-acbc-d288900869bc\" (UID: \"9b7fa7fc-5c26-444b-acbc-d288900869bc\") " Nov 03 08:03:31.629343 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.629322 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7fa7fc-5c26-444b-acbc-d288900869bc-openshift-service-ca" pod "9b7fa7fc-5c26-444b-acbc-d288900869bc" (UID: "9b7fa7fc-5c26-444b-acbc-d288900869bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:03:31.631216 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.631191 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7fa7fc-5c26-444b-acbc-d288900869bc-kube-api-access-5x79p" pod "9b7fa7fc-5c26-444b-acbc-d288900869bc" (UID: "9b7fa7fc-5c26-444b-acbc-d288900869bc"). InnerVolumeSpecName "kube-api-access-5x79p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:03:31.631333 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.631290 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-metrics-certs" pod "9b7fa7fc-5c26-444b-acbc-d288900869bc" (UID: "9b7fa7fc-5c26-444b-acbc-d288900869bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:03:31.631333 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.631316 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-maas-api-tls" pod "9b7fa7fc-5c26-444b-acbc-d288900869bc" (UID: "9b7fa7fc-5c26-444b-acbc-d288900869bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:03:31.730691 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.730661 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7fa7fc-5c26-444b-acbc-d288900869bc-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:31.730691 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.730689 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:31.730892 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.730699 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x79p\" (UniqueName: \"kubernetes.io/projected/9b7fa7fc-5c26-444b-acbc-d288900869bc-kube-api-access-5x79p\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:31.730892 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:31.730751 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7fa7fc-5c26-444b-acbc-d288900869bc-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:32.292374 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:32.292338 4409 generic.go:356] "Generic (PLEG): container finished" podID="9b7fa7fc-5c26-444b-acbc-d288900869bc" containerID="f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733" exitCode=0 Nov 03 08:03:32.292554 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:32.292388 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" Nov 03 08:03:32.292554 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:32.292418 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" event={"ID":"9b7fa7fc-5c26-444b-acbc-d288900869bc","Type":"ContainerDied","Data":"f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733"} Nov 03 08:03:32.292554 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:32.292477 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6" event={"ID":"9b7fa7fc-5c26-444b-acbc-d288900869bc","Type":"ContainerDied","Data":"0f19f3849daf3b9e5e9184c77ab7836426e1c0ed67a921e0b1be0e882a97ee2a"} Nov 03 08:03:32.292554 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:32.292501 4409 scope.go:122] "RemoveContainer" containerID="f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733" Nov 03 08:03:32.302758 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:32.302734 4409 scope.go:122] "RemoveContainer" containerID="f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733" Nov 03 08:03:32.303061 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:03:32.303039 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733\": container with ID starting with f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733 not found: ID does not exist" containerID="f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733" Nov 03 08:03:32.303138 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:32.303068 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733"} err="failed to get container status \"f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733\": rpc error: code = NotFound desc = could not find container \"f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733\": container with ID starting with f9a75cad08fc587bd53594e23cd7c448356d02c50e511b067d43866c1f971733 not found: ID does not exist" Nov 03 08:03:32.470314 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:32.470288 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6"] Nov 03 08:03:32.519440 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:32.519413 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b435d8-d54b89448-6g6g6"] Nov 03 08:03:32.911646 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:32.911600 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7fa7fc-5c26-444b-acbc-d288900869bc" path="/var/lib/kubelet/pods/9b7fa7fc-5c26-444b-acbc-d288900869bc/volumes" Nov 03 08:03:38.774469 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:03:38.774431 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:03:46.107768 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:46.107700 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd"] Nov 03 08:03:46.108167 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:46.107936 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" containerID="cri-o://253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1" gracePeriod=30 Nov 03 08:03:51.017517 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.017484 4409 patch_prober.go:29] interesting pod/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:03:51.017917 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.017560 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:03:51.363792 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.363762 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:03:51.389254 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.389221 4409 generic.go:356] "Generic (PLEG): container finished" podID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerID="253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1" exitCode=0 Nov 03 08:03:51.389384 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.389269 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" Nov 03 08:03:51.389384 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.389307 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" event={"ID":"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4","Type":"ContainerDied","Data":"253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1"} Nov 03 08:03:51.389384 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.389359 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd" event={"ID":"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4","Type":"ContainerDied","Data":"b1dc0d21d4f13d4fe31e10e6e7971fef3080c72a6ed75527cb1de3a8dfa99c80"} Nov 03 08:03:51.389384 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.389381 4409 scope.go:122] "RemoveContainer" containerID="253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1" Nov 03 08:03:51.397970 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.397953 4409 scope.go:122] "RemoveContainer" containerID="30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca" Nov 03 08:03:51.406135 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.406117 4409 scope.go:122] "RemoveContainer" containerID="253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1" Nov 03 08:03:51.406398 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:03:51.406377 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1\": container with ID starting with 253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1 not found: ID does not exist" containerID="253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1" Nov 03 08:03:51.406457 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.406407 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1"} err="failed to get container status \"253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1\": rpc error: code = NotFound desc = could not find container \"253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1\": container with ID starting with 253ecf1237f4e3f1c447e8d343d1cacc6491bbfee707a8e2fe2bef5a7845dcb1 not found: ID does not exist" Nov 03 08:03:51.406457 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.406423 4409 scope.go:122] "RemoveContainer" containerID="30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca" Nov 03 08:03:51.406672 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:03:51.406654 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca\": container with ID starting with 30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca not found: ID does not exist" containerID="30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca" Nov 03 08:03:51.406742 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.406679 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca"} err="failed to get container status \"30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca\": rpc error: code = NotFound desc = could not find container \"30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca\": container with ID starting with 30faaba816217cf582e3437c3b225ef4645ae1e01f6f347100ebc40a3cf4b6ca not found: ID does not exist" Nov 03 08:03:51.522643 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.522621 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-socket\") pod \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " Nov 03 08:03:51.522772 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.522666 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-token\" (UniqueName: \"kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-token\") pod \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " Nov 03 08:03:51.522772 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.522745 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istiod-ca-cert\") pod \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " Nov 03 08:03:51.522772 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.522769 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-certs\") pod \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " Nov 03 08:03:51.522905 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.522820 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-kube-api-access-4rx8n\" (UniqueName: \"kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-kube-api-access-4rx8n\") pod \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " Nov 03 08:03:51.522956 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.522909 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-podinfo\") pod \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " Nov 03 08:03:51.522956 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.522908 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-socket" pod "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" (UID: "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:03:51.523049 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.523000 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-envoy\") pod \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " Nov 03 08:03:51.523109 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.523056 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-data\") pod \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " Nov 03 08:03:51.523172 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.523103 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-certs" pod "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" (UID: "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:03:51.523172 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.523107 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-credential-socket\") pod \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\" (UID: \"68f5a50b-b9c6-4b55-a11a-5b45749b1ba4\") " Nov 03 08:03:51.523172 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.523138 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istiod-ca-cert" pod "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" (UID: "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:03:51.523355 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.523330 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-credential-socket" pod "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" (UID: "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:03:51.523423 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.523391 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-data" pod "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" (UID: "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:03:51.523887 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.523862 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:51.523988 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.523890 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:51.523988 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.523906 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:51.523988 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.523920 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:51.523988 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.523934 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:51.524984 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.524960 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-token" pod "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" (UID: "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:03:51.525101 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.525083 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-kube-api-access-4rx8n" pod "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" (UID: "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4"). InnerVolumeSpecName "kube-api-access-4rx8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:03:51.525186 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.525169 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-podinfo" pod "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" (UID: "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:03:51.525253 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.525233 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-envoy" pod "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" (UID: "68f5a50b-b9c6-4b55-a11a-5b45749b1ba4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:03:51.624489 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.624463 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rx8n\" (UniqueName: \"kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-kube-api-access-4rx8n\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:51.624489 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.624488 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:51.624653 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.624498 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:51.624653 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.624508 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:03:51.813374 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.813342 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd"] Nov 03 08:03:51.821419 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:51.821395 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b435d8-openshift-default-65dcd9b8bf49spd"] Nov 03 08:03:52.911326 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:03:52.911293 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" path="/var/lib/kubelet/pods/68f5a50b-b9c6-4b55-a11a-5b45749b1ba4/volumes" Nov 03 08:04:05.481229 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:04:05.481202 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 08:04:05.485966 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:04:05.485945 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 08:04:08.773625 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:04:08.773589 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:04:38.773959 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:04:38.773918 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:05:08.773625 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:05:08.773587 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:05:14.481924 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:05:14.481864 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-qrckb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:05:14.482798 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:05:14.482761 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-qrckb" podUID="2a99e5d6-cdd0-41f3-804d-5b478c92f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:05:38.774329 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:05:38.774286 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:06:08.773745 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:06:08.773686 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:06:31.779886 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.779855 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x"] Nov 03 08:06:31.780502 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.780437 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9b7fa7fc-5c26-444b-acbc-d288900869bc" containerName="maas-api" Nov 03 08:06:31.780502 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.780456 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7fa7fc-5c26-444b-acbc-d288900869bc" containerName="maas-api" Nov 03 08:06:31.780502 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.780473 4409 cpu_manager.go:414] "containerMap: removing container" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" Nov 03 08:06:31.780502 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.780481 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" Nov 03 08:06:31.780819 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.780621 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" Nov 03 08:06:31.780819 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.780641 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7fa7fc-5c26-444b-acbc-d288900869bc" containerName="maas-api" Nov 03 08:06:31.784107 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.784083 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:31.892192 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.892153 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305c2ec3-2201-4940-8062-1d8afabf3048-metrics-certs\") pod \"maas-api-5bcf779ff9-f7p9x\" (UID: \"305c2ec3-2201-4940-8062-1d8afabf3048\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:31.892192 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.892206 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305c2ec3-2201-4940-8062-1d8afabf3048-maas-api-tls\") pod \"maas-api-5bcf779ff9-f7p9x\" (UID: \"305c2ec3-2201-4940-8062-1d8afabf3048\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:31.892464 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.892251 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcnmt\" (UniqueName: \"kubernetes.io/projected/305c2ec3-2201-4940-8062-1d8afabf3048-kube-api-access-dcnmt\") pod \"maas-api-5bcf779ff9-f7p9x\" (UID: \"305c2ec3-2201-4940-8062-1d8afabf3048\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:31.892464 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.892280 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305c2ec3-2201-4940-8062-1d8afabf3048-openshift-service-ca\") pod \"maas-api-5bcf779ff9-f7p9x\" (UID: \"305c2ec3-2201-4940-8062-1d8afabf3048\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:31.892869 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.892842 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x"] Nov 03 08:06:31.993058 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.993028 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305c2ec3-2201-4940-8062-1d8afabf3048-metrics-certs\") pod \"maas-api-5bcf779ff9-f7p9x\" (UID: \"305c2ec3-2201-4940-8062-1d8afabf3048\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:31.993058 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.993063 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305c2ec3-2201-4940-8062-1d8afabf3048-maas-api-tls\") pod \"maas-api-5bcf779ff9-f7p9x\" (UID: \"305c2ec3-2201-4940-8062-1d8afabf3048\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:31.993267 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.993207 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcnmt\" (UniqueName: \"kubernetes.io/projected/305c2ec3-2201-4940-8062-1d8afabf3048-kube-api-access-dcnmt\") pod \"maas-api-5bcf779ff9-f7p9x\" (UID: \"305c2ec3-2201-4940-8062-1d8afabf3048\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:31.993267 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.993251 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305c2ec3-2201-4940-8062-1d8afabf3048-openshift-service-ca\") pod \"maas-api-5bcf779ff9-f7p9x\" (UID: \"305c2ec3-2201-4940-8062-1d8afabf3048\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:31.994006 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.993982 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305c2ec3-2201-4940-8062-1d8afabf3048-openshift-service-ca\") pod \"maas-api-5bcf779ff9-f7p9x\" (UID: \"305c2ec3-2201-4940-8062-1d8afabf3048\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:31.995703 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.995679 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305c2ec3-2201-4940-8062-1d8afabf3048-metrics-certs\") pod \"maas-api-5bcf779ff9-f7p9x\" (UID: \"305c2ec3-2201-4940-8062-1d8afabf3048\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:31.995988 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:31.995968 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305c2ec3-2201-4940-8062-1d8afabf3048-maas-api-tls\") pod \"maas-api-5bcf779ff9-f7p9x\" (UID: \"305c2ec3-2201-4940-8062-1d8afabf3048\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:32.055697 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:32.055676 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcnmt\" (UniqueName: \"kubernetes.io/projected/305c2ec3-2201-4940-8062-1d8afabf3048-kube-api-access-dcnmt\") pod \"maas-api-5bcf779ff9-f7p9x\" (UID: \"305c2ec3-2201-4940-8062-1d8afabf3048\") " pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:32.097548 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:32.097525 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:32.482246 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:32.482207 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x"] Nov 03 08:06:32.485332 ip-10-0-138-121 kubenswrapper[4409]: W1103 08:06:32.485305 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305c2ec3_2201_4940_8062_1d8afabf3048.slice/crio-7ffeccca93ea73606807ae4920e9e3a4082baa52d0f38b8803b6f83417f29731 WatchSource:0}: Error finding container 7ffeccca93ea73606807ae4920e9e3a4082baa52d0f38b8803b6f83417f29731: Status 404 returned error can't find the container with id 7ffeccca93ea73606807ae4920e9e3a4082baa52d0f38b8803b6f83417f29731 Nov 03 08:06:32.487006 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:32.486982 4409 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:06:33.278322 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:33.278288 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" event={"ID":"305c2ec3-2201-4940-8062-1d8afabf3048","Type":"ContainerStarted","Data":"7ffeccca93ea73606807ae4920e9e3a4082baa52d0f38b8803b6f83417f29731"} Nov 03 08:06:34.285154 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:34.285110 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" event={"ID":"305c2ec3-2201-4940-8062-1d8afabf3048","Type":"ContainerStarted","Data":"43cdf00bbcc60be704d513fe6e5d079eab9c5e3642e02b71414eba6d31513e22"} Nov 03 08:06:34.285560 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:34.285218 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:34.441571 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:34.441512 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" podStartSLOduration=2.613956103 podStartE2EDuration="3.441497107s" totalImagesPullingTime="827.541004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:34.440820833 +0000 UTC m=+4049.633362597" watchObservedRunningTime="2026-11-03 08:06:34.441497107 +0000 UTC m=+4049.634038856" Nov 03 08:06:38.775044 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:06:38.775005 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:06:40.295205 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:40.295174 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bcf779ff9-f7p9x" Nov 03 08:06:40.470836 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:40.470796 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh"] Nov 03 08:06:40.471124 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:40.471099 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" podUID="9255898a-7131-44de-a617-f2c92d75714b" containerName="maas-api" containerID="cri-o://acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b" gracePeriod=30 Nov 03 08:06:41.079600 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.079551 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" podUID="9255898a-7131-44de-a617-f2c92d75714b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.35:8443/health\": dial tcp 10.133.0.35:8443: connect: connection refused" Nov 03 08:06:41.724008 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.723981 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 08:06:41.786309 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.786263 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-metrics-certs\") pod \"9255898a-7131-44de-a617-f2c92d75714b\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " Nov 03 08:06:41.786464 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.786369 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9255898a-7131-44de-a617-f2c92d75714b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9255898a-7131-44de-a617-f2c92d75714b-openshift-service-ca\") pod \"9255898a-7131-44de-a617-f2c92d75714b\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " Nov 03 08:06:41.786464 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.786402 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-maas-api-tls\") pod \"9255898a-7131-44de-a617-f2c92d75714b\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " Nov 03 08:06:41.786464 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.786444 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9255898a-7131-44de-a617-f2c92d75714b-kube-api-access-xrbvz\" (UniqueName: \"kubernetes.io/projected/9255898a-7131-44de-a617-f2c92d75714b-kube-api-access-xrbvz\") pod \"9255898a-7131-44de-a617-f2c92d75714b\" (UID: \"9255898a-7131-44de-a617-f2c92d75714b\") " Nov 03 08:06:41.786816 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.786786 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9255898a-7131-44de-a617-f2c92d75714b-openshift-service-ca" pod "9255898a-7131-44de-a617-f2c92d75714b" (UID: "9255898a-7131-44de-a617-f2c92d75714b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:06:41.788611 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.788587 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-metrics-certs" pod "9255898a-7131-44de-a617-f2c92d75714b" (UID: "9255898a-7131-44de-a617-f2c92d75714b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:06:41.788694 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.788674 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9255898a-7131-44de-a617-f2c92d75714b-kube-api-access-xrbvz" pod "9255898a-7131-44de-a617-f2c92d75714b" (UID: "9255898a-7131-44de-a617-f2c92d75714b"). InnerVolumeSpecName "kube-api-access-xrbvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:06:41.788773 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.788732 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-maas-api-tls" pod "9255898a-7131-44de-a617-f2c92d75714b" (UID: "9255898a-7131-44de-a617-f2c92d75714b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:06:41.887735 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.887633 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9255898a-7131-44de-a617-f2c92d75714b-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:06:41.887735 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.887663 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:06:41.887735 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.887672 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrbvz\" (UniqueName: \"kubernetes.io/projected/9255898a-7131-44de-a617-f2c92d75714b-kube-api-access-xrbvz\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:06:41.887735 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:41.887681 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9255898a-7131-44de-a617-f2c92d75714b-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:06:42.323608 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:42.323568 4409 generic.go:356] "Generic (PLEG): container finished" podID="9255898a-7131-44de-a617-f2c92d75714b" containerID="acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b" exitCode=0 Nov 03 08:06:42.323861 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:42.323614 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" event={"ID":"9255898a-7131-44de-a617-f2c92d75714b","Type":"ContainerDied","Data":"acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b"} Nov 03 08:06:42.323861 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:42.323621 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" Nov 03 08:06:42.323861 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:42.323660 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh" event={"ID":"9255898a-7131-44de-a617-f2c92d75714b","Type":"ContainerDied","Data":"03f63a38ca64f8dc7e8081e77f56fa0ab192d2b6356750a40e97b366827f0466"} Nov 03 08:06:42.323861 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:42.323682 4409 scope.go:122] "RemoveContainer" containerID="acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b" Nov 03 08:06:42.333818 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:42.333792 4409 scope.go:122] "RemoveContainer" containerID="acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b" Nov 03 08:06:42.334112 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:06:42.334093 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b\": container with ID starting with acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b not found: ID does not exist" containerID="acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b" Nov 03 08:06:42.334166 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:42.334124 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b"} err="failed to get container status \"acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b\": rpc error: code = NotFound desc = could not find container \"acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b\": container with ID starting with acf16f690524654fd1c400feaed28d68c6ec3fb9e88e3dc02ed4161e76f9950b not found: ID does not exist" Nov 03 08:06:42.495589 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:42.495554 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh"] Nov 03 08:06:42.545243 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:42.545207 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f9b5fd67-8vnsh"] Nov 03 08:06:42.912156 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:42.912124 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9255898a-7131-44de-a617-f2c92d75714b" path="/var/lib/kubelet/pods/9255898a-7131-44de-a617-f2c92d75714b/volumes" Nov 03 08:06:46.172689 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.172641 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284"] Nov 03 08:06:46.174289 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.174258 4409 cpu_manager.go:414] "containerMap: removing container" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" Nov 03 08:06:46.174448 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.174434 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" Nov 03 08:06:46.174585 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.174574 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9255898a-7131-44de-a617-f2c92d75714b" containerName="maas-api" Nov 03 08:06:46.174816 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.174799 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9255898a-7131-44de-a617-f2c92d75714b" containerName="maas-api" Nov 03 08:06:46.175208 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.175192 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9255898a-7131-44de-a617-f2c92d75714b" containerName="maas-api" Nov 03 08:06:46.175318 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.175307 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f5a50b-b9c6-4b55-a11a-5b45749b1ba4" containerName="istio-proxy" Nov 03 08:06:46.191432 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.191397 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" Nov 03 08:06:46.230247 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.230213 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfc6l\" (UniqueName: \"kubernetes.io/projected/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-kube-api-access-wfc6l\") pod \"payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284\" (UID: \"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc\") " pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" Nov 03 08:06:46.230390 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.230363 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-plugins-config-volume\") pod \"payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284\" (UID: \"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc\") " pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" Nov 03 08:06:46.234459 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.234432 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284"] Nov 03 08:06:46.281471 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.281440 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-16cc64e2\"" Nov 03 08:06:46.281471 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.281441 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-16cc64e2-dockercfg-d98fp\"" Nov 03 08:06:46.331284 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.331254 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-plugins-config-volume\") pod \"payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284\" (UID: \"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc\") " pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" Nov 03 08:06:46.331433 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.331337 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfc6l\" (UniqueName: \"kubernetes.io/projected/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-kube-api-access-wfc6l\") pod \"payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284\" (UID: \"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc\") " pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" Nov 03 08:06:46.332038 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.332019 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-plugins-config-volume\") pod \"payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284\" (UID: \"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc\") " pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" Nov 03 08:06:46.348045 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.348021 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw"] Nov 03 08:06:46.351508 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.351493 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" Nov 03 08:06:46.393913 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.393886 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfc6l\" (UniqueName: \"kubernetes.io/projected/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-kube-api-access-wfc6l\") pod \"payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284\" (UID: \"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc\") " pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" Nov 03 08:06:46.408298 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.408269 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw"] Nov 03 08:06:46.432806 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.432732 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6n9w\" (UniqueName: \"kubernetes.io/projected/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-kube-api-access-w6n9w\") pod \"payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw\" (UID: \"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" Nov 03 08:06:46.432917 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.432838 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw\" (UID: \"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" Nov 03 08:06:46.506939 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.506911 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" Nov 03 08:06:46.534255 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.534216 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6n9w\" (UniqueName: \"kubernetes.io/projected/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-kube-api-access-w6n9w\") pod \"payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw\" (UID: \"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" Nov 03 08:06:46.534377 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.534330 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw\" (UID: \"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" Nov 03 08:06:46.535261 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.535235 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw\" (UID: \"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" Nov 03 08:06:46.597467 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.597436 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6n9w\" (UniqueName: \"kubernetes.io/projected/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-kube-api-access-w6n9w\") pod \"payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw\" (UID: \"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" Nov 03 08:06:46.663540 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.663507 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" Nov 03 08:06:46.883114 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:46.883090 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284"] Nov 03 08:06:46.885480 ip-10-0-138-121 kubenswrapper[4409]: W1103 08:06:46.885450 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdeed67_9d3e_4a98_ac6f_e93fd458f1fc.slice/crio-ec7583604ffe530718dae652ed2212f2578d2dd797610800af45d986c1da84c1 WatchSource:0}: Error finding container ec7583604ffe530718dae652ed2212f2578d2dd797610800af45d986c1da84c1: Status 404 returned error can't find the container with id ec7583604ffe530718dae652ed2212f2578d2dd797610800af45d986c1da84c1 Nov 03 08:06:47.047057 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.047039 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw"] Nov 03 08:06:47.049277 ip-10-0-138-121 kubenswrapper[4409]: W1103 08:06:47.049252 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1e0ec8_f40f_409e_9a8d_813c2f7521ce.slice/crio-bbfbef7801109d6f4bde83bd08676136f5f7cdc4134c8bf550bb32535fbbcda2 WatchSource:0}: Error finding container bbfbef7801109d6f4bde83bd08676136f5f7cdc4134c8bf550bb32535fbbcda2: Status 404 returned error can't find the container with id bbfbef7801109d6f4bde83bd08676136f5f7cdc4134c8bf550bb32535fbbcda2 Nov 03 08:06:47.351033 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.350994 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" event={"ID":"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce","Type":"ContainerStarted","Data":"878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38"} Nov 03 08:06:47.351474 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.351145 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" Nov 03 08:06:47.351474 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.351200 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" event={"ID":"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce","Type":"ContainerStarted","Data":"bbfbef7801109d6f4bde83bd08676136f5f7cdc4134c8bf550bb32535fbbcda2"} Nov 03 08:06:47.352603 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.352583 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" event={"ID":"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc","Type":"ContainerStarted","Data":"96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239"} Nov 03 08:06:47.352728 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.352616 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" event={"ID":"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc","Type":"ContainerStarted","Data":"ec7583604ffe530718dae652ed2212f2578d2dd797610800af45d986c1da84c1"} Nov 03 08:06:47.352776 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.352751 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" Nov 03 08:06:47.469828 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.469571 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" podStartSLOduration=1.469559075 podStartE2EDuration="1.469559075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:47.469510728 +0000 UTC m=+4062.662052480" watchObservedRunningTime="2026-11-03 08:06:47.469559075 +0000 UTC m=+4062.662100825" Nov 03 08:06:47.602281 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.598771 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" podStartSLOduration=1.5987516309999998 podStartE2EDuration="1.598751631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:47.59816539 +0000 UTC m=+4062.790707141" watchObservedRunningTime="2026-11-03 08:06:47.598751631 +0000 UTC m=+4062.791293382" Nov 03 08:06:47.795191 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.795151 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft"] Nov 03 08:06:47.798908 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.798884 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:47.846873 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.846847 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-metrics-certs\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:47.846993 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.846959 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-maas-api-tls\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:47.847049 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.847022 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28qp6\" (UniqueName: \"kubernetes.io/projected/9302c071-297c-4dde-85d3-ced5ecbb2f9f-kube-api-access-28qp6\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:47.847089 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.847052 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9302c071-297c-4dde-85d3-ced5ecbb2f9f-openshift-service-ca\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:47.857952 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.857891 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-16cc64e2-serving-cert\"" Nov 03 08:06:47.859807 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.859777 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft"] Nov 03 08:06:47.948215 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.948186 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-maas-api-tls\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:47.948438 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.948235 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28qp6\" (UniqueName: \"kubernetes.io/projected/9302c071-297c-4dde-85d3-ced5ecbb2f9f-kube-api-access-28qp6\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:47.948438 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.948261 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9302c071-297c-4dde-85d3-ced5ecbb2f9f-openshift-service-ca\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:47.948438 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.948322 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-metrics-certs\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:47.948438 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:06:47.948363 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-serving-cert: secret "maas-api-e2e-ait-16cc64e2-serving-cert" not found Nov 03 08:06:47.948697 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:06:47.948445 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-maas-api-tls podName:9302c071-297c-4dde-85d3-ced5ecbb2f9f nodeName:}" failed. No retries permitted until 2026-11-03 08:06:48.448422733 +0000 UTC m=+4063.640964476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-maas-api-tls") pod "maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" (UID: "9302c071-297c-4dde-85d3-ced5ecbb2f9f") : secret "maas-api-e2e-ait-16cc64e2-serving-cert" not found Nov 03 08:06:47.949083 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.949060 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9302c071-297c-4dde-85d3-ced5ecbb2f9f-openshift-service-ca\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:47.951010 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:47.950984 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-metrics-certs\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:48.010678 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:48.010649 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28qp6\" (UniqueName: \"kubernetes.io/projected/9302c071-297c-4dde-85d3-ced5ecbb2f9f-kube-api-access-28qp6\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:48.454029 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:48.453998 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-maas-api-tls\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:48.456502 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:48.456483 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-maas-api-tls\") pod \"maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:48.712514 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:48.712432 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:49.087917 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:49.087893 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft"] Nov 03 08:06:49.089928 ip-10-0-138-121 kubenswrapper[4409]: W1103 08:06:49.089897 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9302c071_297c_4dde_85d3_ced5ecbb2f9f.slice/crio-f0fafc2415c3cfa0502fa7d14e64280362c62aee71a5d1d0ee9d2e20b774cd3a WatchSource:0}: Error finding container f0fafc2415c3cfa0502fa7d14e64280362c62aee71a5d1d0ee9d2e20b774cd3a: Status 404 returned error can't find the container with id f0fafc2415c3cfa0502fa7d14e64280362c62aee71a5d1d0ee9d2e20b774cd3a Nov 03 08:06:49.365599 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:49.365513 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" event={"ID":"9302c071-297c-4dde-85d3-ced5ecbb2f9f","Type":"ContainerStarted","Data":"f0fafc2415c3cfa0502fa7d14e64280362c62aee71a5d1d0ee9d2e20b774cd3a"} Nov 03 08:06:50.372030 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:50.371994 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" event={"ID":"9302c071-297c-4dde-85d3-ced5ecbb2f9f","Type":"ContainerStarted","Data":"92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083"} Nov 03 08:06:50.372404 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:50.372164 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:50.493263 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:50.493213 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" podStartSLOduration=2.58456565 podStartE2EDuration="3.493200466s" totalImagesPullingTime="908.634816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:50.491878854 +0000 UTC m=+4065.684420607" watchObservedRunningTime="2026-11-03 08:06:50.493200466 +0000 UTC m=+4065.685742215" Nov 03 08:06:56.382752 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:56.382698 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:06:58.360604 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:58.360576 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" Nov 03 08:06:58.361027 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:06:58.360643 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" Nov 03 08:07:08.773776 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:07:08.773742 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:07:29.504752 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.504665 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft"] Nov 03 08:07:29.507310 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.504875 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" podUID="9302c071-297c-4dde-85d3-ced5ecbb2f9f" containerName="maas-api" containerID="cri-o://92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083" gracePeriod=30 Nov 03 08:07:29.789207 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.789183 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:07:29.839751 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.839688 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-maas-api-tls\") pod \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " Nov 03 08:07:29.839932 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.839767 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9302c071-297c-4dde-85d3-ced5ecbb2f9f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9302c071-297c-4dde-85d3-ced5ecbb2f9f-openshift-service-ca\") pod \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " Nov 03 08:07:29.839932 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.839815 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9302c071-297c-4dde-85d3-ced5ecbb2f9f-kube-api-access-28qp6\" (UniqueName: \"kubernetes.io/projected/9302c071-297c-4dde-85d3-ced5ecbb2f9f-kube-api-access-28qp6\") pod \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " Nov 03 08:07:29.839932 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.839844 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-metrics-certs\") pod \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\" (UID: \"9302c071-297c-4dde-85d3-ced5ecbb2f9f\") " Nov 03 08:07:29.840129 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.840102 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9302c071-297c-4dde-85d3-ced5ecbb2f9f-openshift-service-ca" pod "9302c071-297c-4dde-85d3-ced5ecbb2f9f" (UID: "9302c071-297c-4dde-85d3-ced5ecbb2f9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:07:29.842141 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.842113 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9302c071-297c-4dde-85d3-ced5ecbb2f9f-kube-api-access-28qp6" pod "9302c071-297c-4dde-85d3-ced5ecbb2f9f" (UID: "9302c071-297c-4dde-85d3-ced5ecbb2f9f"). InnerVolumeSpecName "kube-api-access-28qp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:07:29.842141 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.842116 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-metrics-certs" pod "9302c071-297c-4dde-85d3-ced5ecbb2f9f" (UID: "9302c071-297c-4dde-85d3-ced5ecbb2f9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:07:29.842274 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.842137 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-maas-api-tls" pod "9302c071-297c-4dde-85d3-ced5ecbb2f9f" (UID: "9302c071-297c-4dde-85d3-ced5ecbb2f9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:07:29.941728 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.941683 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9302c071-297c-4dde-85d3-ced5ecbb2f9f-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:07:29.941728 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.941734 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28qp6\" (UniqueName: \"kubernetes.io/projected/9302c071-297c-4dde-85d3-ced5ecbb2f9f-kube-api-access-28qp6\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:07:29.941969 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.941752 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:07:29.941969 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.941766 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9302c071-297c-4dde-85d3-ced5ecbb2f9f-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:07:29.988001 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:29.987971 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284"] Nov 03 08:07:30.044851 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:30.044777 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw"] Nov 03 08:07:30.567207 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:30.567176 4409 generic.go:356] "Generic (PLEG): container finished" podID="9302c071-297c-4dde-85d3-ced5ecbb2f9f" containerID="92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083" exitCode=0 Nov 03 08:07:30.567652 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:30.567217 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" Nov 03 08:07:30.567652 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:30.567266 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" event={"ID":"9302c071-297c-4dde-85d3-ced5ecbb2f9f","Type":"ContainerDied","Data":"92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083"} Nov 03 08:07:30.567652 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:30.567311 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft" event={"ID":"9302c071-297c-4dde-85d3-ced5ecbb2f9f","Type":"ContainerDied","Data":"f0fafc2415c3cfa0502fa7d14e64280362c62aee71a5d1d0ee9d2e20b774cd3a"} Nov 03 08:07:30.567652 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:30.567336 4409 scope.go:122] "RemoveContainer" containerID="92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083" Nov 03 08:07:30.577686 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:30.577667 4409 scope.go:122] "RemoveContainer" containerID="92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083" Nov 03 08:07:30.577994 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:07:30.577975 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083\": container with ID starting with 92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083 not found: ID does not exist" containerID="92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083" Nov 03 08:07:30.578053 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:30.578003 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083"} err="failed to get container status \"92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083\": rpc error: code = NotFound desc = could not find container \"92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083\": container with ID starting with 92fd631a9956bc2da39f063c5ae115671bd8e248a0eae0b270b193bcb8940083 not found: ID does not exist" Nov 03 08:07:30.733692 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:30.733660 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft"] Nov 03 08:07:30.783414 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:30.783385 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16cc64e2-68d4bb97bf-7nvft"] Nov 03 08:07:30.911497 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:30.911422 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9302c071-297c-4dde-85d3-ced5ecbb2f9f" path="/var/lib/kubelet/pods/9302c071-297c-4dde-85d3-ced5ecbb2f9f/volumes" Nov 03 08:07:34.989047 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:34.989009 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" podUID="cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc" containerName="payload-processing" containerID="cri-o://96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239" gracePeriod=25 Nov 03 08:07:35.045621 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.045579 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" podUID="9e1e0ec8-f40f-409e-9a8d-813c2f7521ce" containerName="payload-pre-processing" containerID="cri-o://878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38" gracePeriod=25 Nov 03 08:07:35.291289 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.291261 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" Nov 03 08:07:35.316668 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.316647 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" Nov 03 08:07:35.393586 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.393553 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-kube-api-access-w6n9w\" (UniqueName: \"kubernetes.io/projected/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-kube-api-access-w6n9w\") pod \"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce\" (UID: \"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce\") " Nov 03 08:07:35.393700 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.393605 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-plugins-config-volume\") pod \"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc\" (UID: \"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc\") " Nov 03 08:07:35.393700 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.393687 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-kube-api-access-wfc6l\" (UniqueName: \"kubernetes.io/projected/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-kube-api-access-wfc6l\") pod \"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc\" (UID: \"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc\") " Nov 03 08:07:35.393827 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.393772 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-plugins-config-volume\") pod \"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce\" (UID: \"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce\") " Nov 03 08:07:35.394179 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.394019 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-plugins-config-volume" pod "cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc" (UID: "cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:07:35.394179 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.394112 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:07:35.394760 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.394732 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-plugins-config-volume" pod "9e1e0ec8-f40f-409e-9a8d-813c2f7521ce" (UID: "9e1e0ec8-f40f-409e-9a8d-813c2f7521ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:07:35.396079 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.396058 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-kube-api-access-wfc6l" pod "cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc" (UID: "cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc"). InnerVolumeSpecName "kube-api-access-wfc6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:07:35.396159 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.396097 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-kube-api-access-w6n9w" pod "9e1e0ec8-f40f-409e-9a8d-813c2f7521ce" (UID: "9e1e0ec8-f40f-409e-9a8d-813c2f7521ce"). InnerVolumeSpecName "kube-api-access-w6n9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:07:35.495059 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.495030 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6n9w\" (UniqueName: \"kubernetes.io/projected/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-kube-api-access-w6n9w\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:07:35.495059 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.495056 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfc6l\" (UniqueName: \"kubernetes.io/projected/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc-kube-api-access-wfc6l\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:07:35.495059 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.495065 4409 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce-plugins-config-volume\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:07:35.595002 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.594969 4409 generic.go:356] "Generic (PLEG): container finished" podID="9e1e0ec8-f40f-409e-9a8d-813c2f7521ce" containerID="878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38" exitCode=0 Nov 03 08:07:35.595177 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.595015 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" Nov 03 08:07:35.595177 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.595056 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" event={"ID":"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce","Type":"ContainerDied","Data":"878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38"} Nov 03 08:07:35.595177 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.595108 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw" event={"ID":"9e1e0ec8-f40f-409e-9a8d-813c2f7521ce","Type":"ContainerDied","Data":"bbfbef7801109d6f4bde83bd08676136f5f7cdc4134c8bf550bb32535fbbcda2"} Nov 03 08:07:35.595177 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.595135 4409 scope.go:122] "RemoveContainer" containerID="878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38" Nov 03 08:07:35.596596 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.596574 4409 generic.go:356] "Generic (PLEG): container finished" podID="cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc" containerID="96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239" exitCode=0 Nov 03 08:07:35.596749 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.596608 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" event={"ID":"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc","Type":"ContainerDied","Data":"96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239"} Nov 03 08:07:35.596749 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.596633 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" Nov 03 08:07:35.596749 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.596660 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284" event={"ID":"cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc","Type":"ContainerDied","Data":"ec7583604ffe530718dae652ed2212f2578d2dd797610800af45d986c1da84c1"} Nov 03 08:07:35.606425 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.606408 4409 scope.go:122] "RemoveContainer" containerID="878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38" Nov 03 08:07:35.606771 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:07:35.606744 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38\": container with ID starting with 878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38 not found: ID does not exist" containerID="878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38" Nov 03 08:07:35.606860 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.606783 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38"} err="failed to get container status \"878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38\": rpc error: code = NotFound desc = could not find container \"878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38\": container with ID starting with 878a2b19d7cefe2a270492fe6d60613e114330d816138ec00aad038c5c6dcb38 not found: ID does not exist" Nov 03 08:07:35.606860 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.606809 4409 scope.go:122] "RemoveContainer" containerID="96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239" Nov 03 08:07:35.615101 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.615081 4409 scope.go:122] "RemoveContainer" containerID="96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239" Nov 03 08:07:35.615389 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:07:35.615371 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239\": container with ID starting with 96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239 not found: ID does not exist" containerID="96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239" Nov 03 08:07:35.615440 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.615395 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239"} err="failed to get container status \"96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239\": rpc error: code = NotFound desc = could not find container \"96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239\": container with ID starting with 96fe7859237bfd8c624cdd332593cbeb3c322879d192e16f28ee761ed32dc239 not found: ID does not exist" Nov 03 08:07:35.761734 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.761687 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw"] Nov 03 08:07:35.767928 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.767908 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-16cc64e2-7cd4cd4b7-wpfsw"] Nov 03 08:07:35.878973 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.878917 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284"] Nov 03 08:07:35.884913 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:35.884893 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16cc64e2-5664d9975f-8k284"] Nov 03 08:07:36.911595 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:36.911563 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1e0ec8-f40f-409e-9a8d-813c2f7521ce" path="/var/lib/kubelet/pods/9e1e0ec8-f40f-409e-9a8d-813c2f7521ce/volumes" Nov 03 08:07:36.912064 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:36.912044 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc" path="/var/lib/kubelet/pods/cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc/volumes" Nov 03 08:07:38.774741 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:07:38.774676 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:07:41.587977 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.587946 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x"] Nov 03 08:07:41.588393 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.588375 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9e1e0ec8-f40f-409e-9a8d-813c2f7521ce" containerName="payload-pre-processing" Nov 03 08:07:41.588393 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.588387 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1e0ec8-f40f-409e-9a8d-813c2f7521ce" containerName="payload-pre-processing" Nov 03 08:07:41.588469 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.588400 4409 cpu_manager.go:414] "containerMap: removing container" podUID="cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc" containerName="payload-processing" Nov 03 08:07:41.588469 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.588405 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc" containerName="payload-processing" Nov 03 08:07:41.588469 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.588419 4409 cpu_manager.go:414] "containerMap: removing container" podUID="9302c071-297c-4dde-85d3-ced5ecbb2f9f" containerName="maas-api" Nov 03 08:07:41.588469 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.588424 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9302c071-297c-4dde-85d3-ced5ecbb2f9f" containerName="maas-api" Nov 03 08:07:41.588605 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.588499 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1e0ec8-f40f-409e-9a8d-813c2f7521ce" containerName="payload-pre-processing" Nov 03 08:07:41.588605 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.588508 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbdeed67-9d3e-4a98-ac6f-e93fd458f1fc" containerName="payload-processing" Nov 03 08:07:41.588605 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.588516 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="9302c071-297c-4dde-85d3-ced5ecbb2f9f" containerName="maas-api" Nov 03 08:07:41.593166 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.593139 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.653171 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.653134 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-3ef065-openshift-default-dockercfg-n4bkm\"" Nov 03 08:07:41.658150 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.657726 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jdj\" (UniqueName: \"kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-kube-api-access-s7jdj\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.658150 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.657807 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-certs\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.658150 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.658007 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-token\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.658150 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.658045 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-data\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.658150 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.658076 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee110f57-5005-43cb-aa75-792df60d6f3a-istiod-ca-cert\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.658150 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.658142 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-podinfo\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.658503 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.658170 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-socket\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.658503 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.658207 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-envoy\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.658503 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.658256 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-credential-socket\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.702644 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.702602 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x"] Nov 03 08:07:41.759590 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.759540 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-podinfo\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.759887 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.759864 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-socket\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.760029 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.760013 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-envoy\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.760172 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.760156 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-credential-socket\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.760352 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.760335 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jdj\" (UniqueName: \"kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-kube-api-access-s7jdj\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.760463 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.760442 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-socket\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.760562 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.760547 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-certs\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.760745 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.760727 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-token\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.761036 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.761005 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-data\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.761169 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.761152 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee110f57-5005-43cb-aa75-792df60d6f3a-istiod-ca-cert\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.772063 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.763484 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-certs\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.772063 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.763791 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-credential-socket\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.772063 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.764531 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-data\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.772063 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.765274 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-envoy\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.772063 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.765654 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee110f57-5005-43cb-aa75-792df60d6f3a-istiod-ca-cert\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.772063 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.766943 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-podinfo\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.827824 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.827790 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jdj\" (UniqueName: \"kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-kube-api-access-s7jdj\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.827967 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.827870 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-token\") pod \"e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:41.906793 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:41.906669 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:42.337837 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:42.337805 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x"] Nov 03 08:07:42.340558 ip-10-0-138-121 kubenswrapper[4409]: W1103 08:07:42.340518 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee110f57_5005_43cb_aa75_792df60d6f3a.slice/crio-aee70e06bc18f0c28efccf43928f08861e2dd43e002b3da9f185f67fdf6426b6 WatchSource:0}: Error finding container aee70e06bc18f0c28efccf43928f08861e2dd43e002b3da9f185f67fdf6426b6: Status 404 returned error can't find the container with id aee70e06bc18f0c28efccf43928f08861e2dd43e002b3da9f185f67fdf6426b6 Nov 03 08:07:42.343294 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:42.343261 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:07:42.343387 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:42.343335 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:07:42.343387 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:42.343366 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:07:42.649568 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:42.649481 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" event={"ID":"ee110f57-5005-43cb-aa75-792df60d6f3a","Type":"ContainerStarted","Data":"eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96"} Nov 03 08:07:42.649568 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:42.649549 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" event={"ID":"ee110f57-5005-43cb-aa75-792df60d6f3a","Type":"ContainerStarted","Data":"aee70e06bc18f0c28efccf43928f08861e2dd43e002b3da9f185f67fdf6426b6"} Nov 03 08:07:42.773233 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:42.773181 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" podStartSLOduration=1.77316661 podStartE2EDuration="1.77316661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:42.771423191 +0000 UTC m=+4117.963964979" watchObservedRunningTime="2026-11-03 08:07:42.77316661 +0000 UTC m=+4117.965708359" Nov 03 08:07:42.915728 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:42.915633 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:43.912434 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:43.912400 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:44.658878 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:44.658841 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:44.659801 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:44.659782 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:07:51.840557 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:51.840521 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7"] Nov 03 08:07:51.844928 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:51.844910 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:51.899052 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:51.899027 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-3ef065-serving-cert\"" Nov 03 08:07:51.949157 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:51.949132 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7"] Nov 03 08:07:51.995951 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:51.995919 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c64c869-5011-4af7-bf9d-50e07067e6db-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:51.996272 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:51.996248 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-metrics-certs\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:51.996363 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:51.996287 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:51.996363 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:51.996334 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rqw\" (UniqueName: \"kubernetes.io/projected/1c64c869-5011-4af7-bf9d-50e07067e6db-kube-api-access-76rqw\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:52.103080 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:52.102980 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rqw\" (UniqueName: \"kubernetes.io/projected/1c64c869-5011-4af7-bf9d-50e07067e6db-kube-api-access-76rqw\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:52.103894 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:52.103857 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c64c869-5011-4af7-bf9d-50e07067e6db-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:52.104049 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:52.104033 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-metrics-certs\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:52.104114 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:52.104059 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:52.104238 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:07:52.104222 4409 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-serving-cert: secret "maas-api-e2e-authcfg-http-3ef065-serving-cert" not found Nov 03 08:07:52.104309 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:07:52.104293 4409 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-maas-api-tls podName:1c64c869-5011-4af7-bf9d-50e07067e6db nodeName:}" failed. No retries permitted until 2026-11-03 08:07:52.604270148 +0000 UTC m=+4127.796811877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-maas-api-tls") pod "maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" (UID: "1c64c869-5011-4af7-bf9d-50e07067e6db") : secret "maas-api-e2e-authcfg-http-3ef065-serving-cert" not found Nov 03 08:07:52.105401 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:52.105336 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c64c869-5011-4af7-bf9d-50e07067e6db-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:52.113121 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:52.113091 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-metrics-certs\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:52.166347 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:52.166320 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rqw\" (UniqueName: \"kubernetes.io/projected/1c64c869-5011-4af7-bf9d-50e07067e6db-kube-api-access-76rqw\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:52.609540 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:52.609506 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:52.612151 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:52.612123 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:52.758072 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:52.758043 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:53.131758 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:53.131737 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7"] Nov 03 08:07:53.134034 ip-10-0-138-121 kubenswrapper[4409]: W1103 08:07:53.134009 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c64c869_5011_4af7_bf9d_50e07067e6db.slice/crio-b3508a99ad9d06dd0dab16b3fb6f4c8d73917995dd7a4b5d8f2d93df00050532 WatchSource:0}: Error finding container b3508a99ad9d06dd0dab16b3fb6f4c8d73917995dd7a4b5d8f2d93df00050532: Status 404 returned error can't find the container with id b3508a99ad9d06dd0dab16b3fb6f4c8d73917995dd7a4b5d8f2d93df00050532 Nov 03 08:07:53.706276 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:53.706232 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" event={"ID":"1c64c869-5011-4af7-bf9d-50e07067e6db","Type":"ContainerStarted","Data":"b3508a99ad9d06dd0dab16b3fb6f4c8d73917995dd7a4b5d8f2d93df00050532"} Nov 03 08:07:54.713950 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:54.713914 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" event={"ID":"1c64c869-5011-4af7-bf9d-50e07067e6db","Type":"ContainerStarted","Data":"699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39"} Nov 03 08:07:54.714362 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:54.714022 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:07:54.834240 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:07:54.834189 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" podStartSLOduration=2.85691423 podStartE2EDuration="3.834170812s" totalImagesPullingTime="977.256582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:54.833696607 +0000 UTC m=+4130.026238359" watchObservedRunningTime="2026-11-03 08:07:54.834170812 +0000 UTC m=+4130.026712562" Nov 03 08:08:00.725297 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:00.725270 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:08:08.777939 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:08:08.777892 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:08:09.821313 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:09.821281 4409 generic.go:356] "Generic (PLEG): container finished" podID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerID="eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96" exitCode=0 Nov 03 08:08:09.821756 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:09.821350 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" event={"ID":"ee110f57-5005-43cb-aa75-792df60d6f3a","Type":"ContainerDied","Data":"eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96"} Nov 03 08:08:09.821756 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:09.821669 4409 scope.go:122] "RemoveContainer" containerID="eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96" Nov 03 08:08:09.822811 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:09.822780 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:09.822912 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:09.822886 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:09.822967 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:09.822936 4409 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 03 08:08:10.827594 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:10.827555 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" event={"ID":"ee110f57-5005-43cb-aa75-792df60d6f3a","Type":"ContainerStarted","Data":"e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e"} Nov 03 08:08:10.908925 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:10.908892 4409 patch_prober.go:29] interesting pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 08:08:10.909091 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:10.908944 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 08:08:10.912038 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:10.912017 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:08:11.907399 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:11.907366 4409 patch_prober.go:29] interesting pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 08:08:11.907836 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:11.907418 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 08:08:12.907020 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:12.906985 4409 patch_prober.go:29] interesting pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 08:08:12.907216 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:12.907034 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 08:08:13.907286 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:13.907249 4409 patch_prober.go:29] interesting pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 08:08:13.907738 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:13.907305 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 08:08:14.659016 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:14.658983 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:08:14.908998 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:14.908944 4409 patch_prober.go:29] interesting pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 03 08:08:14.909526 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:14.909029 4409 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 03 08:08:15.973490 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:15.973453 4409 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:08:15.976094 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:08:15.976074 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:08:38.778069 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:08:38.778025 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:09:05.517549 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:05.517514 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 08:09:05.520163 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:05.520137 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4ffc_3378427f-b215-4a67-8a9d-aab1f8b05d8f/ovn-acl-logging/0.log" Nov 03 08:09:08.773451 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:09:08.773410 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:09:38.774175 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:09:38.774134 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:09:50.271796 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.271760 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7"] Nov 03 08:09:50.272229 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.271945 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" podUID="1c64c869-5011-4af7-bf9d-50e07067e6db" containerName="maas-api" containerID="cri-o://699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39" gracePeriod=30 Nov 03 08:09:50.529461 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.529392 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:09:50.671387 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.671353 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c64c869-5011-4af7-bf9d-50e07067e6db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c64c869-5011-4af7-bf9d-50e07067e6db-openshift-service-ca\") pod \"1c64c869-5011-4af7-bf9d-50e07067e6db\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " Nov 03 08:09:50.671595 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.671442 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c64c869-5011-4af7-bf9d-50e07067e6db-kube-api-access-76rqw\" (UniqueName: \"kubernetes.io/projected/1c64c869-5011-4af7-bf9d-50e07067e6db-kube-api-access-76rqw\") pod \"1c64c869-5011-4af7-bf9d-50e07067e6db\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " Nov 03 08:09:50.671595 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.671493 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-maas-api-tls\") pod \"1c64c869-5011-4af7-bf9d-50e07067e6db\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " Nov 03 08:09:50.671595 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.671546 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-metrics-certs\") pod \"1c64c869-5011-4af7-bf9d-50e07067e6db\" (UID: \"1c64c869-5011-4af7-bf9d-50e07067e6db\") " Nov 03 08:09:50.671885 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.671811 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c64c869-5011-4af7-bf9d-50e07067e6db-openshift-service-ca" pod "1c64c869-5011-4af7-bf9d-50e07067e6db" (UID: "1c64c869-5011-4af7-bf9d-50e07067e6db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:09:50.672209 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.672162 4409 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c64c869-5011-4af7-bf9d-50e07067e6db-openshift-service-ca\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:09:50.676183 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.676134 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-metrics-certs" pod "1c64c869-5011-4af7-bf9d-50e07067e6db" (UID: "1c64c869-5011-4af7-bf9d-50e07067e6db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:09:50.681917 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.681867 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c64c869-5011-4af7-bf9d-50e07067e6db-kube-api-access-76rqw" pod "1c64c869-5011-4af7-bf9d-50e07067e6db" (UID: "1c64c869-5011-4af7-bf9d-50e07067e6db"). InnerVolumeSpecName "kube-api-access-76rqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:09:50.682242 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.682213 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-maas-api-tls" pod "1c64c869-5011-4af7-bf9d-50e07067e6db" (UID: "1c64c869-5011-4af7-bf9d-50e07067e6db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:09:50.773261 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.773226 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76rqw\" (UniqueName: \"kubernetes.io/projected/1c64c869-5011-4af7-bf9d-50e07067e6db-kube-api-access-76rqw\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:09:50.773261 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.773257 4409 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-maas-api-tls\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:09:50.773261 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:50.773266 4409 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c64c869-5011-4af7-bf9d-50e07067e6db-metrics-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:09:51.331758 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:51.331695 4409 generic.go:356] "Generic (PLEG): container finished" podID="1c64c869-5011-4af7-bf9d-50e07067e6db" containerID="699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39" exitCode=0 Nov 03 08:09:51.332223 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:51.331769 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" Nov 03 08:09:51.332223 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:51.331770 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" event={"ID":"1c64c869-5011-4af7-bf9d-50e07067e6db","Type":"ContainerDied","Data":"699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39"} Nov 03 08:09:51.332223 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:51.331903 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7" event={"ID":"1c64c869-5011-4af7-bf9d-50e07067e6db","Type":"ContainerDied","Data":"b3508a99ad9d06dd0dab16b3fb6f4c8d73917995dd7a4b5d8f2d93df00050532"} Nov 03 08:09:51.332223 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:51.331935 4409 scope.go:122] "RemoveContainer" containerID="699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39" Nov 03 08:09:51.340872 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:51.340841 4409 scope.go:122] "RemoveContainer" containerID="699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39" Nov 03 08:09:51.341155 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:09:51.341133 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39\": container with ID starting with 699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39 not found: ID does not exist" containerID="699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39" Nov 03 08:09:51.341256 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:51.341160 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39"} err="failed to get container status \"699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39\": rpc error: code = NotFound desc = could not find container \"699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39\": container with ID starting with 699e9e916e95a9f3eaa3dccdb338c047be57c22230ddf9fd7117f743b1c49f39 not found: ID does not exist" Nov 03 08:09:51.499657 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:51.499629 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7"] Nov 03 08:09:51.549143 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:51.549116 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-3ef065-757897dfc9-9t4w7"] Nov 03 08:09:52.912626 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:52.912589 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c64c869-5011-4af7-bf9d-50e07067e6db" path="/var/lib/kubelet/pods/1c64c869-5011-4af7-bf9d-50e07067e6db/volumes" Nov 03 08:09:59.762066 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:59.762036 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x"] Nov 03 08:09:59.762447 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:09:59.762234 4409 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerName="istio-proxy" containerID="cri-o://e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e" gracePeriod=30 Nov 03 08:10:00.978237 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:00.978202 4409 patch_prober.go:29] interesting pod/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 08:10:00.978770 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:00.978270 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:10:05.010751 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.010695 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:10:05.105796 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.105770 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-socket\") pod \"ee110f57-5005-43cb-aa75-792df60d6f3a\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " Nov 03 08:10:05.105955 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.105826 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-podinfo\") pod \"ee110f57-5005-43cb-aa75-792df60d6f3a\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " Nov 03 08:10:05.105955 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.105862 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-kube-api-access-s7jdj\" (UniqueName: \"kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-kube-api-access-s7jdj\") pod \"ee110f57-5005-43cb-aa75-792df60d6f3a\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " Nov 03 08:10:05.105955 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.105898 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-envoy\") pod \"ee110f57-5005-43cb-aa75-792df60d6f3a\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " Nov 03 08:10:05.105955 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.105918 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-token\" (UniqueName: \"kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-token\") pod \"ee110f57-5005-43cb-aa75-792df60d6f3a\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " Nov 03 08:10:05.106183 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.105987 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee110f57-5005-43cb-aa75-792df60d6f3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee110f57-5005-43cb-aa75-792df60d6f3a-istiod-ca-cert\") pod \"ee110f57-5005-43cb-aa75-792df60d6f3a\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " Nov 03 08:10:05.106183 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106021 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-credential-socket\") pod \"ee110f57-5005-43cb-aa75-792df60d6f3a\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " Nov 03 08:10:05.106183 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106048 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-certs\") pod \"ee110f57-5005-43cb-aa75-792df60d6f3a\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " Nov 03 08:10:05.106183 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106043 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-socket" pod "ee110f57-5005-43cb-aa75-792df60d6f3a" (UID: "ee110f57-5005-43cb-aa75-792df60d6f3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:10:05.106183 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106148 4409 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-data\") pod \"ee110f57-5005-43cb-aa75-792df60d6f3a\" (UID: \"ee110f57-5005-43cb-aa75-792df60d6f3a\") " Nov 03 08:10:05.106448 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106341 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-certs" pod "ee110f57-5005-43cb-aa75-792df60d6f3a" (UID: "ee110f57-5005-43cb-aa75-792df60d6f3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:10:05.106448 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106361 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-credential-socket" pod "ee110f57-5005-43cb-aa75-792df60d6f3a" (UID: "ee110f57-5005-43cb-aa75-792df60d6f3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:10:05.106555 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106517 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-data" pod "ee110f57-5005-43cb-aa75-792df60d6f3a" (UID: "ee110f57-5005-43cb-aa75-792df60d6f3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:10:05.106616 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106561 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee110f57-5005-43cb-aa75-792df60d6f3a-istiod-ca-cert" pod "ee110f57-5005-43cb-aa75-792df60d6f3a" (UID: "ee110f57-5005-43cb-aa75-792df60d6f3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:10:05.106893 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106872 4409 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:10:05.106974 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106899 4409 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee110f57-5005-43cb-aa75-792df60d6f3a-istiod-ca-cert\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:10:05.106974 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106914 4409 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-credential-socket\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:10:05.106974 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106930 4409 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-workload-certs\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:10:05.106974 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.106944 4409 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-data\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:10:05.108208 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.108181 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-token" pod "ee110f57-5005-43cb-aa75-792df60d6f3a" (UID: "ee110f57-5005-43cb-aa75-792df60d6f3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:10:05.108339 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.108280 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-podinfo" pod "ee110f57-5005-43cb-aa75-792df60d6f3a" (UID: "ee110f57-5005-43cb-aa75-792df60d6f3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:10:05.108414 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.108389 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-kube-api-access-s7jdj" pod "ee110f57-5005-43cb-aa75-792df60d6f3a" (UID: "ee110f57-5005-43cb-aa75-792df60d6f3a"). InnerVolumeSpecName "kube-api-access-s7jdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:10:05.108414 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.108393 4409 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-envoy" pod "ee110f57-5005-43cb-aa75-792df60d6f3a" (UID: "ee110f57-5005-43cb-aa75-792df60d6f3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:10:05.207609 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.207575 4409 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-podinfo\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:10:05.207609 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.207612 4409 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7jdj\" (UniqueName: \"kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-kube-api-access-s7jdj\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:10:05.207851 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.207630 4409 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-envoy\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:10:05.207851 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.207646 4409 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee110f57-5005-43cb-aa75-792df60d6f3a-istio-token\") on node \"ip-10-0-138-121.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:10:05.413453 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.413368 4409 generic.go:356] "Generic (PLEG): container finished" podID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerID="e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e" exitCode=0 Nov 03 08:10:05.413453 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.413422 4409 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" Nov 03 08:10:05.413632 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.413460 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" event={"ID":"ee110f57-5005-43cb-aa75-792df60d6f3a","Type":"ContainerDied","Data":"e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e"} Nov 03 08:10:05.413632 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.413515 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x" event={"ID":"ee110f57-5005-43cb-aa75-792df60d6f3a","Type":"ContainerDied","Data":"aee70e06bc18f0c28efccf43928f08861e2dd43e002b3da9f185f67fdf6426b6"} Nov 03 08:10:05.413632 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.413539 4409 scope.go:122] "RemoveContainer" containerID="e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e" Nov 03 08:10:05.423081 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.423066 4409 scope.go:122] "RemoveContainer" containerID="eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96" Nov 03 08:10:05.431535 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.431517 4409 scope.go:122] "RemoveContainer" containerID="e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e" Nov 03 08:10:05.431802 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:10:05.431780 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e\": container with ID starting with e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e not found: ID does not exist" containerID="e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e" Nov 03 08:10:05.431872 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.431810 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e"} err="failed to get container status \"e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e\": rpc error: code = NotFound desc = could not find container \"e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e\": container with ID starting with e58901ee63c3d2ac9ce11c1b93a6130fd0f9043b97401a62c2fe12e820c0669e not found: ID does not exist" Nov 03 08:10:05.431872 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.431827 4409 scope.go:122] "RemoveContainer" containerID="eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96" Nov 03 08:10:05.432071 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:10:05.432052 4409 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96\": container with ID starting with eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96 not found: ID does not exist" containerID="eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96" Nov 03 08:10:05.432134 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.432082 4409 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96"} err="failed to get container status \"eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96\": rpc error: code = NotFound desc = could not find container \"eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96\": container with ID starting with eb248d2d8bdab2c3e95203a1711f5ca9d54b5caa2b6c49d22bb9334b2ffdee96 not found: ID does not exist" Nov 03 08:10:05.584363 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.584340 4409 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x"] Nov 03 08:10:05.590373 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:05.590347 4409 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-3ef065-openshift-default-644d5db5cc-fcx2x"] Nov 03 08:10:06.911619 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:06.911582 4409 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" path="/var/lib/kubelet/pods/ee110f57-5005-43cb-aa75-792df60d6f3a/volumes" Nov 03 08:10:08.774053 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:10:08.774020 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:10:24.692153 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:24.690803 4409 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f66wc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:10:24.693391 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:10:24.693362 4409 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f66wc" podUID="56224d4f-9ba8-4f4d-a938-0e5dbc6c6504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:10:38.774113 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:10:38.774068 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:11:08.774786 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:11:08.774746 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:11:15.300240 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:15.300202 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f56845b76-pf2td_ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf/authorino/0.log" Nov 03 08:11:27.270976 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:27.270945 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-76r6q_f552b885-fd66-4001-a396-326521f15680/manager/0.log" Nov 03 08:11:27.858871 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:27.858831 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d848cbdf4-6dpvx_a490802e-9166-438d-b140-60fe7ced2b9a/manager/0.log" Nov 03 08:11:28.224080 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:28.223992 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lb7g2_5bb50aec-86da-4891-a86f-5217b41dbd40/server/0.log" Nov 03 08:11:28.462307 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:28.462273 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ttjfs_cbec41ca-ada9-4e0b-9753-25988976e073/manager/0.log" Nov 03 08:11:28.750599 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:28.750574 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7pvz9_784a166b-28e5-4550-b5ab-0dad59624f54/proxy/0.log" Nov 03 08:11:29.794987 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:29.794956 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bcf779ff9-f7p9x_305c2ec3-2201-4940-8062-1d8afabf3048/maas-api/0.log" Nov 03 08:11:30.734220 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:30.734189 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-hgbzc_4887bd60-5f24-41a7-814c-9a84fadb3126/postgres/0.log" Nov 03 08:11:32.286975 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:32.286945 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f56845b76-pf2td_ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf/authorino/0.log" Nov 03 08:11:35.112212 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:35.112174 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c8rtx_f7410ef1-cc43-4ca1-b173-546497e70c1a/discovery/0.log" Nov 03 08:11:35.363670 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:35.363596 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86b9d545b-j69hh_5d8f88ce-ff40-439d-ba00-1881218b47f6/kube-auth-proxy/0.log" Nov 03 08:11:36.283382 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:36.283350 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-f66wc_56224d4f-9ba8-4f4d-a938-0e5dbc6c6504/istio-proxy/0.log" Nov 03 08:11:37.231115 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:37.231087 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-qrckb_2a99e5d6-cdd0-41f3-804d-5b478c92f432/istio-proxy/0.log" Nov 03 08:11:37.681577 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:37.681547 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-tq7bl_599908f3-c725-49f4-9151-5e780af03db6/istio-proxy/0.log" Nov 03 08:11:38.634381 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:38.634347 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hn2nn_aafe8fd2-50e9-4eea-8b6b-e686f1242127/payload-processing/0.log" Nov 03 08:11:38.774210 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:11:38.774165 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:11:39.733756 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:39.733706 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz_37837181-c211-4404-846d-4b15159fac0c/main/0.log" Nov 03 08:11:39.983081 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:39.983044 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-wl597_cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1/main/0.log" Nov 03 08:11:40.936390 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:40.936357 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-sckng_fb400850-a2b2-4b94-839f-88e16402cf5f/main/0.log" Nov 03 08:11:41.170016 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:41.169968 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6_6b36e4ad-8060-4a62-b8c2-7468072396ae/main/0.log" Nov 03 08:11:42.366466 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:42.366430 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f56845b76-pf2td_ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf/authorino/0.log" Nov 03 08:11:53.320215 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:53.320116 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-76r6q_f552b885-fd66-4001-a396-326521f15680/manager/0.log" Nov 03 08:11:53.895751 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:53.895702 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d848cbdf4-6dpvx_a490802e-9166-438d-b140-60fe7ced2b9a/manager/0.log" Nov 03 08:11:54.160471 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:54.160349 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lb7g2_5bb50aec-86da-4891-a86f-5217b41dbd40/server/0.log" Nov 03 08:11:54.390632 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:54.390604 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ttjfs_cbec41ca-ada9-4e0b-9753-25988976e073/manager/0.log" Nov 03 08:11:54.710945 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:54.710912 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7pvz9_784a166b-28e5-4550-b5ab-0dad59624f54/proxy/0.log" Nov 03 08:11:55.546254 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:55.546197 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bcf779ff9-f7p9x_305c2ec3-2201-4940-8062-1d8afabf3048/maas-api/0.log" Nov 03 08:11:56.591065 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:56.591039 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-hgbzc_4887bd60-5f24-41a7-814c-9a84fadb3126/postgres/0.log" Nov 03 08:11:57.813325 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:11:57.813299 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f56845b76-pf2td_ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf/authorino/0.log" Nov 03 08:12:00.369914 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:00.369886 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c8rtx_f7410ef1-cc43-4ca1-b173-546497e70c1a/discovery/0.log" Nov 03 08:12:00.621132 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:00.621055 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86b9d545b-j69hh_5d8f88ce-ff40-439d-ba00-1881218b47f6/kube-auth-proxy/0.log" Nov 03 08:12:01.527997 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:01.527965 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-f66wc_56224d4f-9ba8-4f4d-a938-0e5dbc6c6504/istio-proxy/0.log" Nov 03 08:12:02.459640 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:02.459606 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-qrckb_2a99e5d6-cdd0-41f3-804d-5b478c92f432/istio-proxy/0.log" Nov 03 08:12:02.939361 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:02.939329 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-tq7bl_599908f3-c725-49f4-9151-5e780af03db6/istio-proxy/0.log" Nov 03 08:12:03.877962 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:03.877935 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hn2nn_aafe8fd2-50e9-4eea-8b6b-e686f1242127/payload-processing/0.log" Nov 03 08:12:04.727252 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:04.727220 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-nx8hz_37837181-c211-4404-846d-4b15159fac0c/main/0.log" Nov 03 08:12:04.957320 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:04.957286 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-wl597_cfc3575a-55c9-4bb0-a6df-eb8ff497b5f1/main/0.log" Nov 03 08:12:06.024936 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:06.024904 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-sckng_fb400850-a2b2-4b94-839f-88e16402cf5f/main/0.log" Nov 03 08:12:06.308485 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:06.308460 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-ftkf6_6b36e4ad-8060-4a62-b8c2-7468072396ae/main/0.log" Nov 03 08:12:08.773339 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:12:08.773291 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:12:36.392793 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:36.392756 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d848cbdf4-6dpvx_a490802e-9166-438d-b140-60fe7ced2b9a/manager/0.log" Nov 03 08:12:36.974756 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:36.974729 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d848cbdf4-6dpvx_a490802e-9166-438d-b140-60fe7ced2b9a/manager/0.log" Nov 03 08:12:37.745489 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:37.745461 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hn2nn_aafe8fd2-50e9-4eea-8b6b-e686f1242127/payload-processing/0.log" Nov 03 08:12:37.976819 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:12:37.976793 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hn2nn_aafe8fd2-50e9-4eea-8b6b-e686f1242127/payload-processing/0.log" Nov 03 08:12:38.775050 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:12:38.775016 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:13:08.773481 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:13:08.773447 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:13:16.127078 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:16.127043 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d848cbdf4-6dpvx_a490802e-9166-438d-b140-60fe7ced2b9a/manager/0.log" Nov 03 08:13:16.717582 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:16.717555 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d848cbdf4-6dpvx_a490802e-9166-438d-b140-60fe7ced2b9a/manager/0.log" Nov 03 08:13:17.514987 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:17.514959 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hn2nn_aafe8fd2-50e9-4eea-8b6b-e686f1242127/payload-processing/0.log" Nov 03 08:13:17.772573 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:17.772500 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hn2nn_aafe8fd2-50e9-4eea-8b6b-e686f1242127/payload-processing/0.log" Nov 03 08:13:28.324982 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:28.324944 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zhpcz_8d5f4689-033d-4aa8-8603-a56cbdd449cb/global-pull-secret-syncer/0.log" Nov 03 08:13:31.092261 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:31.092228 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nj9qb_eff68601-5319-4dc8-93c6-16de197bd812/konnectivity-agent/0.log" Nov 03 08:13:32.513856 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:32.513819 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-121.us-west-1.compute.internal_305af933eb9c6382ff5eabda70a436a6/haproxy/0.log" Nov 03 08:13:36.437696 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:36.437662 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f56845b76-pf2td_ee04ce1d-4bd5-4d30-880f-ae8c4a34ebcf/authorino/0.log" Nov 03 08:13:37.366611 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.366569 4409 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4"] Nov 03 08:13:37.367081 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.367063 4409 cpu_manager.go:414] "containerMap: removing container" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerName="istio-proxy" Nov 03 08:13:37.367185 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.367083 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerName="istio-proxy" Nov 03 08:13:37.367185 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.367099 4409 cpu_manager.go:414] "containerMap: removing container" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerName="istio-proxy" Nov 03 08:13:37.367185 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.367108 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerName="istio-proxy" Nov 03 08:13:37.367185 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.367126 4409 cpu_manager.go:414] "containerMap: removing container" podUID="1c64c869-5011-4af7-bf9d-50e07067e6db" containerName="maas-api" Nov 03 08:13:37.367185 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.367133 4409 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c64c869-5011-4af7-bf9d-50e07067e6db" containerName="maas-api" Nov 03 08:13:37.367445 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.367266 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee110f57-5005-43cb-aa75-792df60d6f3a" containerName="istio-proxy" Nov 03 08:13:37.367445 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.367282 4409 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c64c869-5011-4af7-bf9d-50e07067e6db" containerName="maas-api" Nov 03 08:13:37.370757 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.370735 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.386541 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.386515 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21df1ff5-ba98-43e1-b5e7-eb74ca014604-lib-modules\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.386668 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.386554 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21df1ff5-ba98-43e1-b5e7-eb74ca014604-sys\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.386668 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.386573 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21df1ff5-ba98-43e1-b5e7-eb74ca014604-proc\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.386773 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.386704 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn2qg\" (UniqueName: \"kubernetes.io/projected/21df1ff5-ba98-43e1-b5e7-eb74ca014604-kube-api-access-hn2qg\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.386816 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.386772 4409 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21df1ff5-ba98-43e1-b5e7-eb74ca014604-podres\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.428767 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.428707 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9lpwd\"/\"kube-root-ca.crt\"" Nov 03 08:13:37.430184 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.430161 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4"] Nov 03 08:13:37.477163 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.477134 4409 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9lpwd\"/\"openshift-service-ca.crt\"" Nov 03 08:13:37.477554 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.477204 4409 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9lpwd\"/\"default-dockercfg-blmhb\"" Nov 03 08:13:37.487747 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.487700 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21df1ff5-ba98-43e1-b5e7-eb74ca014604-lib-modules\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.487848 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.487758 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21df1ff5-ba98-43e1-b5e7-eb74ca014604-sys\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.487848 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.487776 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21df1ff5-ba98-43e1-b5e7-eb74ca014604-proc\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.487848 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.487809 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn2qg\" (UniqueName: \"kubernetes.io/projected/21df1ff5-ba98-43e1-b5e7-eb74ca014604-kube-api-access-hn2qg\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.487848 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.487836 4409 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21df1ff5-ba98-43e1-b5e7-eb74ca014604-podres\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.487848 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.487839 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21df1ff5-ba98-43e1-b5e7-eb74ca014604-sys\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.488017 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.487843 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21df1ff5-ba98-43e1-b5e7-eb74ca014604-lib-modules\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.488017 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.487907 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21df1ff5-ba98-43e1-b5e7-eb74ca014604-proc\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.488017 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.487955 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21df1ff5-ba98-43e1-b5e7-eb74ca014604-podres\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.551139 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.551105 4409 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn2qg\" (UniqueName: \"kubernetes.io/projected/21df1ff5-ba98-43e1-b5e7-eb74ca014604-kube-api-access-hn2qg\") pod \"perf-node-gather-daemonset-lwfp4\" (UID: \"21df1ff5-ba98-43e1-b5e7-eb74ca014604\") " pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:37.683762 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:37.683631 4409 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:38.062143 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:38.062114 4409 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4"] Nov 03 08:13:38.064367 ip-10-0-138-121 kubenswrapper[4409]: W1103 08:13:38.064326 4409 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod21df1ff5_ba98_43e1_b5e7_eb74ca014604.slice/crio-30073b137c9af44390e8aeaeb172de1588935e0757d3fd678cd564f3976dfb1e WatchSource:0}: Error finding container 30073b137c9af44390e8aeaeb172de1588935e0757d3fd678cd564f3976dfb1e: Status 404 returned error can't find the container with id 30073b137c9af44390e8aeaeb172de1588935e0757d3fd678cd564f3976dfb1e Nov 03 08:13:38.445878 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:38.445761 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" event={"ID":"21df1ff5-ba98-43e1-b5e7-eb74ca014604","Type":"ContainerStarted","Data":"e0224cc5d33ab7c7a64ee5b53b20827d04241daea602788abad260f84f82ad58"} Nov 03 08:13:38.445878 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:38.445871 4409 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" event={"ID":"21df1ff5-ba98-43e1-b5e7-eb74ca014604","Type":"ContainerStarted","Data":"30073b137c9af44390e8aeaeb172de1588935e0757d3fd678cd564f3976dfb1e"} Nov 03 08:13:38.446149 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:38.445898 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:38.563695 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:38.563639 4409 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" podStartSLOduration=1.5636228989999998 podStartE2EDuration="1.563622899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:38.562571216 +0000 UTC m=+4473.755112969" watchObservedRunningTime="2026-11-03 08:13:38.563622899 +0000 UTC m=+4473.756164697" Nov 03 08:13:38.774658 ip-10-0-138-121 kubenswrapper[4409]: E1103 08:13:38.774581 4409 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqbfp" containerName="extractor" Nov 03 08:13:41.569196 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:41.569166 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06bbffc1-4081-4059-8b5c-40fb6567a736/alertmanager/0.log" Nov 03 08:13:41.992550 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:41.992462 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06bbffc1-4081-4059-8b5c-40fb6567a736/config-reloader/0.log" Nov 03 08:13:42.413069 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:42.413041 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06bbffc1-4081-4059-8b5c-40fb6567a736/kube-rbac-proxy-web/0.log" Nov 03 08:13:42.836041 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:42.835985 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06bbffc1-4081-4059-8b5c-40fb6567a736/kube-rbac-proxy/0.log" Nov 03 08:13:43.257446 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:43.257369 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06bbffc1-4081-4059-8b5c-40fb6567a736/kube-rbac-proxy-metric/0.log" Nov 03 08:13:43.678450 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:43.678419 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06bbffc1-4081-4059-8b5c-40fb6567a736/prom-label-proxy/0.log" Nov 03 08:13:44.099664 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:44.099631 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06bbffc1-4081-4059-8b5c-40fb6567a736/init-config-reloader/0.log" Nov 03 08:13:44.462759 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:44.462668 4409 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9lpwd/perf-node-gather-daemonset-lwfp4" Nov 03 08:13:49.615962 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:49.615928 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s92rs_8d325d30-1335-4213-9d57-d86d25726e89/node-exporter/0.log" Nov 03 08:13:50.044361 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:50.044290 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s92rs_8d325d30-1335-4213-9d57-d86d25726e89/kube-rbac-proxy/0.log" Nov 03 08:13:50.463123 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:50.463098 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s92rs_8d325d30-1335-4213-9d57-d86d25726e89/init-textfile/0.log" Nov 03 08:13:50.895704 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:50.895674 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ms6g9_8628283a-b865-41bb-9126-b6bb140650d3/kube-rbac-proxy-main/0.log" Nov 03 08:13:51.315779 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:51.315747 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ms6g9_8628283a-b865-41bb-9126-b6bb140650d3/kube-rbac-proxy-self/0.log" Nov 03 08:13:51.733757 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:51.733671 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-ms6g9_8628283a-b865-41bb-9126-b6bb140650d3/openshift-state-metrics/0.log" Nov 03 08:13:55.163543 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:55.163516 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-mchvh_82cb9f11-c15b-4ae1-885a-9474727d453b/prometheus-operator/0.log" Nov 03 08:13:55.579856 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:55.579828 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-mchvh_82cb9f11-c15b-4ae1-885a-9474727d453b/kube-rbac-proxy/0.log" Nov 03 08:13:56.430179 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:56.430145 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8649454f66-sldzs_1a26273e-e854-4a0b-8ef4-95a792060cf3/telemeter-client/0.log" Nov 03 08:13:56.848940 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:56.848901 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8649454f66-sldzs_1a26273e-e854-4a0b-8ef4-95a792060cf3/reload/0.log" Nov 03 08:13:57.269170 ip-10-0-138-121 kubenswrapper[4409]: I1103 08:13:57.269104 4409 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8649454f66-sldzs_1a26273e-e854-4a0b-8ef4-95a792060cf3/kube-rbac-proxy/0.log"